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PREFACE 

This  Note  oresents  one  of  seven  cese  studies  developed  as  part  of  the  Project  AIR 
FORCE  study  “Managing  Risks  in  Weapon  Systems  Development  Projects,”  which  has 
developed  case  studies  on  the  level,  distribution,  and  risk  in  the  range  of  major  U.S.  Air 
Force  development  programs  conducted  primarily  during  the  1980s.  Notes  based  on  those 
case  studies  offer  concise  descriptions  and  analyses  of  the  policies  used  by  the  Air  Force  to 
manage  and  distribute  risk.  They  are  aimed  primarily  at  high-level  government  officials 
concerned  with  the  management  of  research  and  development,  including  senior  Air  Force 
staff,  senior  Department  of  Defense  (DoD)  officials,  and  congressional  staff.  The  Notes 
should  also  be  useful  to  policy  analysts  concerned  with  the  management  of  large-scale 
research  and  development,  especially  in  DoD. 

This  Note  examines  risks  associated  with  the  program  used  to  develop  the  new  FIOO- 
PW-220  and  FllO-GE-100  derivative  fighter  engines.  The  derivative  development  described 
here  has  important  implications  for  future  policies  that  could  place  greater  emphasis  on  such 
development  to  facilitate  quick  and  flexible  responses  to  unexpected  changes  in  a  diffuse 
threat.  Information  is  current  as  of  summer  1991. 

Other  Notes  written  in  this  project  include  the  following: 

•  S.  J.  Bodilly,  Case  Study  of  Risk  Management  in  the  USAF  B -IB  Bomber 
Program  ,  RAND,  N-3616-AF,  1993. 

•  S.  J.  Bodilly,  Case  Study  of  Risk  Management  in  the  USAF  LANTIRN  Program, 
RAND,  N-3617-AF,  1993. 

•  F.  Camm,  The  F-16  Multinational  Staged  Improvement  Program:  A  Case  Study 
of  Risk  Assessment  and  Risk  Management ,  RAND,  N-3619-AF,  1993. 

•  K.  R.  Mayer,  The  Development  of  the  Advanced  Medium  Range  AirtoAir  Missile: 
A  Case  Study  of  Risk  and  Reward  in  Weapon  System  Acquisition,  RAND,  N-3620- 
AF,  forthcoming. 

Two  unpublished  papers  also  have  been  written  by  T.  J.  Webb  on  risk  management  during 
the  development  of  the  Global  Positioning  System  Block  I  satellite  and  risk  managerae.it  in 
preparing  for  development  of  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint 
STARS).  A  summary  of  these  Notes  and  papers  and  the  policy  conclusions  based  on  them  is 
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found  in  T.  K.  Glennan  et  al.,  Barriers  to  Managing  Risk  in  Large-Scale  Weapon  System 
Development  Programs,  RAND,  MR-248-AF,  forthcoming, 

The  Air  Force  sponsor  for  the  study  is  the  Deputy  Assistant  Secretary  of  the  Air  Force 
(Contracting)  (SAF/AQC).  The  work  has  been  conducted  in  the  Resource  Management 
Program  of  Project  AIR  FORCE,  The  principal  investigator  at  RAND  is  Dr.  Thomas  K. 
Glennan. 


-  V  - 


SUMMARY 


The  “Great  Engine  War”  pitted  Pratt  and  Whitney  (P&W)  and  General  Electric  (GE) 
against  one  another  to  supply  engines,  the  FlOO-PW-220  and  the  FllO-GE-lOO,  respectively, 
for  the  Air  Force’s  new  F-15  and  F-16  fighters.  Known  more  formally  as  the  Alternate 
Fighter  Engine  competition,  this  acquisition  used  “derivative*  engines — engines  that 
incorporated  small  changes  in  selected  parts  of  existing  engines  to  greatly  improve 
operability,  durability,  and  the  operating  and  support  costs  of  fighter  engines.  This  Note 
examines  the  exceptionally  successful  development  programs  that  created  these  engine^  ;iiifl 
seeks  the  basis  for  that  success,  giving  special  attention  to  risk  assessment  and  risk 
management  in  the  two  development  programs.  This  summary  briefly  reviews  the  history  of 
the  two  developments,  raises  four  basic  policy  issues  revealed  by  our  analysis  to  be 
important,  and  then  reviews  briefly  several  lessons  offered  by  these  developments  for  the 
future. 

A  BRIEF  HISTORY 

These  engines  were  effectively  a  response  to  the  Air  Force’s  operational  experience 
with  the  P&W  FlOO-PW-lOO  and  -200  engines,  used  by  the  Air  Force  to  power  its  F-15  and 
F-16  fighters,  respectively.  The  FlOO  met  its  initial  and  primary  design  goal  of  doubling  the 
thrust-to-weight  ratio  for  operational  fighter  engines.  But,  following  deployment  in  1974,  it 
experienced  serious  difficulties,  two  of  which  were  significant:  “stall-stagnation"  and 
extremely  short  life  with  high  maintenance  costs.  Stall-stagnation  occurred  under  certain 
operating  conditions,  requiring  the  pilot  to  shut  down  and  restart  the  engine  in  flight.  It 
presented  a  danger  in  the  two-engine  F-15  and  a  serious  threat  to  safety  in  the  single-engine 
F-16.  The  engine’s  extremely  short  lifetime — the  period  between  depot  overhauls — and  high 
maintenance  requirements  drove  up  its  operating  costs.  As  these  problems  came  to  light,  the 
Air  Force  tried  to  induce  P&W  to  resolve  them.  The  Air  Force  and  P&W  could  not  agree  on 
who  should  pay  to  fix  the  problems,  end  the  relationship  between  them  deteriorated  as  the 
problems  persisted  in  the  fleet,  forcing  the  Air  Force  to  restrict  operations  and  pay  high 
maintenance  costs. 

During  the  same  period,  GE  faced  a  different  problem.  It  had  lost  to  P&W  in  the 
competitions  to  provide  engines  for  the  F-14,  F’-15,  and  F-16,  which  effectively  locked  ir,  out  of 
the  U.S.  market  for  fighter  engines.  Seeking  a  way  back  in,  in  1975  GE  used  its  own  funds  to 
develop  a  demonstrator  engine,  the  FIOIX,  that  it  hoped  to  use  as  the  basis  of  an  engine  to 


persuade  the  Navy  to  reengine  its  F-14s.  The  Navy  was  not,  interested,  but  the  Air  Force 
was.  It  was  seeking  a  low-risk  engine  to  use  in  its  experimental  Engine  Model  Derivative 
Program  (EMDP).  The  Air  Force  also  saw  the  FIOIX  as  the  basis  for  a  possible  alternative  to 
the  FlOO,  and  a  potential  threat  that  it  could  use  to  obtain  better  performar.^,;  f/om  P&W.  In 
1979,  the  Air  Force  used  its  EMDP  to  fund  a  limited  development  program  based  on  the 
FlOlX,  the  FlOl  Derivative  Fighter  Engine  (DFE)  program.  At  the  same  time,  it  also  began 
a  much  smaller  FlOO  EMDP  effort  to  enhance  the  performance  of  the  P&W  FlOO. 

Pratt  and  Whitney  intensely  resisted  this  threat  to  its  monopoly  position  with  the  Air 
Force,  but  the  Air  Force  maintained  its  support  for  the  GE  engine.  The  FlOl  DFE  EMDP 
demonstrated  that  the  GE  engine  could  eliminate  stall-stagnations  and  dramatically  improve 
durability  and  operating  costs;  the  new  engine  directly  (and  quite  deliberately)  met  the  Air 
Force’s  principal  concerns  about  the  FlOO.  These  results  provided  the  basis  for  an  Air  Force 
decision  to  develop  an  engine  that  could  formally  compete  with  the  FlOO  for  future 
production.  In  1982,  the  Air  Force  initiated  a  full-scale  development  based  on  the  FlOl  DFE 
to  verify  what  it  now  called  the  FllO-GE-lOO  engine,  for  use  in  future,  newly  produced  F-15 
and  F-16  fighters.  At  the  same  time,  it  initiated  full-scale  development  of  three  modifications 
to  the  FlOO  engine.  Drawing  on  those  modifications  and  a  fourth  development  effort  also 
under  way  at  the  same  time,  P&W  drew  up  specifications  for  an  improved  FlOO,  the  FlOO- 
PW-220,  that  it  could  offer  to  compete  against  the  FllO-GE-lOO  for  future  production. 

These  two  engines  provided  the  basis  for  the  Alternate  Fighter  Engine  competition. 

The  Air  Force  held  the  production  competition  in  1983  and,  in  1984,  decided  to  buy  engines 
from  both  contractors.  Full-scale  development  continued,  achieving  product  verification  for 
both  engines  in  1985  and  delivering  the  first  engines  to  the  field  in  1986.  Although  typical 
problems  arose,  both  developments  yielded  engines  for  deplo>Tnent  that  operated  as 
expected,  on  schedule,  and  without  cost  overruns. 

ISSUES  FOR  ANALYSIS 

Why  were  these  developments  so  successful?  What  risks  did  they  present  and  how 
were  those  risks  assessed  and  managed?  In  this  Note,  we  view  risk  as  a  source  of 
uncertainty  that  creates  the  possibility  of  a  negative  outcome;  the  worse  the  outcome  or  the 
larger  its  probability  is,  the  larger  is  the  risk  involved.  All  participants  in  these 
developments  agree  that  the  assessment  and  management  of  such  risk  are  central  to  the 
management  of  an  engine  development.  By  definition,  a  sj’stem  development  systematical'y 
reduces  a  set  of  technological  risks — the  possibility  that  the  development  will  be  unable  to 
achieve  the  technological  goals  set  for  it — over  time.  Over  the  course  of  such  a  development, 


its  managers  face  another  set  of  risks,  from  inside  and  outside  a  program,  that  must  be 
addressed  continually  by  the  development  manager  in  the  normal  course  of  conducting  the 
program.  Such  risks  concern  especially  the  level  of  political  support  and  funding  for  the 
development,  tiie  requirements  set  for  the  development,  the  cost  and  schedule  for  the 
development  itself,  and  economic  factors  affecting  the  cost  of  the  system  that  is  finally 
developed. 

In  the  subject  developments,  four  factors  were  especially  important  to  the  assessment 
and  management  of  risk: 

•  Low  level  of  technological  risk  posed  by  tlie  derivative  developments  undertaken. 

•  Use  of  competition  to  motivate  the  contractors  involved. 

•  Experience  of  the  Propulsion  SPO  overseeing  the  developments, 

•  Nature  of  the  contracts  used  to  structure  the  developments. 

Each  of  these  factors  contributed  to  the  success  of  the  two  developments. 

Low  Inherent  Risks 

With  one  exception,  the  developments  did  not  present  serious  risks  for  the  Air  Force  or 
its  contractors.  The  exception  resulted  from  the  presence  of  competition,  which  we  discuss 
next. 

Technologically,  both  developments  relied  heavily  on  existing  technology.  Using 
different  approaches,  GE  and  P&W  discovered  that  they  could  resolve  the  principal  problems 
with  the  FIDO  engine  by  making  adjustments  in  existing  engines.  GE  started  with  the 
engine  it  had  developed  for  the  B-1,  the  FlOl,  and  scaled  portions  of  it  for  a  fighter.  P&W 
changed  parts  of  its  FlOO  engine  that  were  selected  specifically  to  resolve  the  problems  at 
hand.  Hence,  neither  approach  posed  major  technological  risks. 

Furthermore,  the  developers  did  not  face  serious  risks  from  outside  the  programs. 
Higher  level  Air  Force  officials  raised  initial  questions  about  funding,  and  the  Air  Force 
conducted  the  initial  limited  developments  as  spare,  “success-oriented”  developments.  But  in 
the  end,  the  development  managers  were  able,  with  minor  exceptions,  to  conduct  the 
developments  on  their  own  terms.  They  did  not  need  to  respond  seriously  to  risks  created  by 
unexpected  changes  in  policy  outside  the  developments,  changes  that  might  have  affected  the 
support  for  or  requirements  of  the  developments. 
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Compotition 

The  presence  of  competition  clearly  increased  risk  for  P&W  by  threatening  its  status 
as  the  dominant  fighter-engine  supplier  to  the  Air  Force.  It  also  increased  GE's  uncertainty, 
but  in  a  positive  way,  creating  an  opportunity  for  GE  to  parlay  its  small  investment  in  the 
FIOIX  into  a  large  stake  in  the  fighter-engine  market.  Sudi  competition  reduced  the  risk 
faced  by  the  Air  Force  by  giving  it  options;  If  P&W  did  not  become  more  responsive  to  its 
needs,  the  Air  Force  could  favor  GE  in  production  buys,  and  that  threat  was  likely  to  make 
P&W  more  responsive.  For  example,  if  the  Air  Force  contracted  only  with  GE  or  P&W,  the 
contractor  could  easily  exploit  any  surprises — unanticipated  events — to  reopen  its  ag^reement 
with  the  Air  Force  and  adjust  the  agreement  to  make  its  terms  more  favorable  to  the 
contractor.  Competition  between  two  contractors  enhanced  the  Air  Force’s  power  in  the 
situation,  limiting  the  negative  consequences  if  either  contractor  reopened  the  agreement 
unexpectedly.  Each  contractor  knew  that  the  Air  Force  could  credibly  threaten  to  counter 
such  an  action  by  favoring  the  other  contractor  in  future  production  buys. 

Many  participants  suggest  that  the  key  to  the  competition’s  success  was  the  warranty 
that  accompanied  the  production  contracts.  That  warranty  forced  both  contractors  to  fairly 
clearly  reveal  their  expectations  about  the  actual  performance  (broadly  defined)  of  their 
engines  following  deployment  And  future  performance  lay  at  the  heart  of  the 
developments.^  Such  information  in  itself  reduced  the  Air  Force’s  risk,  and  the  warranty’s 
terms  defined  a  formal  approach  to  sharing  risks  in  the  future. 

One  would  expect  the  incentives  created  by  warranties  to  improve  the  engines  created 
by  these  developments,  but  participants  in  the  developments  disagree.  They  believe  that  the 
engine  developers  created  the  same  designs  they  would  have  created  in  the  absence  of  the 
warranties.  To  the  extent  that  warranties  affected  the  developments  at  all,  they  did  so  only 
by  improving  the  information  available  during  the  competition  and  thereby  enhancing  the 
efficacy  of  the  competition.  - 

Experienced  Developers 

The  Propulsion  SPO  was  formed  in  1977  as  a  focal  point  for  solving  previous  problems 
in  engine  development.  Its  managers  came  to  that  task  with  considerable  experience  in 
developing  engines  and  a  common  vision  of  their  task — ^to  bring  better  empirical  data  to  bear 
on  engine  development,  to  take  advantage  of  new  formal  methods  for  assessing  risks 

iln  the  engine  community,  performance  refers  primarily  to  the  thrust  or  thrust-to-weight  ratio 
of  an  engine.  In  the  broader  acquisition  community,  performance  includes  all  aspects  of  a  system’s 
technical  characteristics.  For  en^nes,  performance  includes  thrust,  operability,  durability, 
supportability,  and  reliability.  Unless  otherwise  stated,  we  use  the  broader  definition  in  this  Note. 


-  IX  - 


associated  with  the  durability  and  supportability  of  engines,  and  ultimately  to  field  engines 
only  after  developers  had  demonstrated  emd  verified  that  they  would  be  sufficiently  durable, 
operable,  and  supportable.  This  vision  matched  the  subject  developments  well,  and  the 
developers  brought  considerable  experience  to  bear  as  they  mounted  the  developments. 

Many  aspects  of  that  experience  were  important.  The  key  managers  in  the  two  engine 
developments  came  to  the  SPO  with  ;onsiderable  experience  and  familiarity  with  engine- 
related  technologies.  Because  the  SPO  managed  a  number  of  developments,  it  could  continue 
to  accumulate  experience  and  compare  experience  across  developments  more  easily  than 
SPOs  responsible  for  only  one  development.  Such  experience  in  particular  increased  the 
SPO’s  understanding  of  its  principal  contractors,  who  were  also  the  contractors  responsible 
for  the  developments;  enhanced  the  SPO’s  working  relationship  with  its  principal  test 
facilities;  and  allowed  rapid  application  and  refinement  of  new  risk  assessment  techniques. 
When  such  developments  were  initiated,  the  standard  procedures  to  which  the  SPO  would 
attribute  its  general  success  with  new  engines  were  already  in  place.  Together,  these  factors 
gave  the  SPO  an  ability  to  plan  and  coordinate  fairly  complex  developments  and  to  react 
confidently  to  surprises.  This  ability  and  confidence  probably  reduced  the  risk  associated 
with  unexpected  events,  allowing  the  SPO  to  react  flexibly  and  creatively  to  .surprises  that 
might  have  stymied  a  less  experienced  staff. 

Contracts 

The  two  developments  proceeded  under  many  separate  but  coordinated  contracts.  One 
aspect  of  the  FlOO-PW-220  development  occurred  under  a  cost-plus-award-fee  contract;  all 
other  development  occurred  under  firm-fixed-price  contracts.  Although  many  current 
observers  believe  that  firm-fixed-price  contracts  impose  excessive  risks  on  contractors  in 
development  programs,  GE  and  F&W  readily  accepted  such  arrangements  and  condurtpd  the 
developments  satisfactorily  despite  risks  that  such  contracts  might  have  imposed. 

Such  contracts  could  be  written  in  part  because  the  developments  did  not  pose  Isu'ge 
risks  for  the  contractors.  The  full  risks  the  contractors  associated  with  the  production 
competition  would  be  felt  in  the  future.  But  other  aspects  of  the  contracts  also  limited  the 
risks  faced  by  the  contractors.  The  contracts’  statements  of  work  tended  to  focus  on  the 
co.'duct  of  development,  not  its  products.  Such  attention  is  consistent  with  a  general 
obser\'ation  among  developers  that  the  way  a  contract  is  monitored  can  easily  be  more 
important  than  how  it  is  written.  It  also  reflects  the  Propulsion  SPO’s  reputation  as  a  hands- 
on  organization  that  worked  more  closely  with  its  contractors  than  most  SPOs.  In  addition, 
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special  clauses  provided  well-articuiated  means  of  sharing  important  risks  associated  svith 
specific  problems  of  funding  and  test  failure. 

LESSONS  FOR  OTHER  DEVELOPMENTS 

What  can  be  learned  from  these  developments  that  might  be  applied  elsewhere?  We 
must  be  cautious  about  drawing  firm  conclusions  based  on  such  a  limited  set  of 
developments.  But  information  on  these  two  developments  is  consistent  with  a  number  of 
hypotheses. 

Derivative  development  can  reduce  risk  in  limited  circumstances.  The  Air 
Force  uses  derivative  engine  developments  when  those  developments  allow  the  Air  Force  to 
achieve  its  development  goals  But  derivative  developments  do  not  allow  large  advances  in 
broadly  conceived  system  performance.  Large  advances  require  more  traditional, 

“centerline”  engine  developoients,  which  inherently  entail  mere  technological  risk  than  that 
observed  here. 

Continuity  in  SPO  management  significantly  enhances  risk  management.  As 
a  standing  SPO  whose  existence  extends  beyond  that  of  individual  development  programs, 
the  Propulsion  SPO  has  accumulated  experience,  doctrine,  procedures,  and  knowledge  about 
its  contractors,  all  of  which  contribute  significantly  to  its  ability  to  anticipate  and  manage 
problems.  These  sources  of  order  and  stability,  perhaps  ironically,  gave  the  SPO  the  skill 
and  confidence  required  to  react  flexibly  and  creatively  to  unexpected  events  in  these 
developments.  Analogous  statements  apply  to  the  contractors  themselves. 

Formal  risk  assessment  plays  a  limited  but  important  role  in  general  risk 
management.  New  concepts  and  methods  developed  in  the  1970s,  including  the  accelerated 
mission  test,  the  four-step  development  process,  and  the  engine  structural  integrity  program, 
gave  the  two  engine  developers  tools  to  manage  numerous  important  risks.  Although  such 
tools  enabled  the  developers  to  achieve  the  durability  and  supportability  goals  sought  for  the 
two  engines,  they  do  not  help  managers  deal  with  broader,  less-defined  risks.  Here,  there 
remains  no  substitute  for  the  judgment  of  experienced  managers  familiar  with  the  routine 
management  of  risk  in  system  developments. 

Higher  level  cognizance  increases  some  program  risks  and  reduces  others. 
The  general  absence  of  higher  level  interest  in  these  developments  allowed  their  immediate 
managers  to  set  realistic  performance,  schedule,  and  cost  goals  and  ultimately  to  meet  those 
goals.  In  this  sense,  the  lack  of  interest  reduced  the  external  risk  experienced  in  the  two 
developments.  But  that  same  lack  of  interest  also  left  the  developments  without  high-level 
advocates  to  protect  the  programs’  funding,  increasing  external  risk.  Such  risks  must  be 
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balanced.  One  way  to  reduce  this  conflict  is  high-level  oversight  that  recognizes  the 
importance  of  setting  and  achieving  realistic  goals  and  of  actively  challenging  goals  that  do 
not  appear  realistic. 

Competition  redistributes  risk  in  a  development  program.  Competition 
increases  the  risk  perceived  by  the  contractors  and  j  educes  the  risk  perceived  by  the  Air 
Force.  At  any  time,  the  Air  Force  can  change  the  market  shares  of  its  contractors,  increasing 
their  risks  and,  by  so  doing,  can  reduce  its  dependence  on  the  poorer  performer,  thereby 
reducing  its  own  risk.  Given  the  price  the  Air  Force  pays  its  contractors  for  contracts, 
competition  also  increases  the  risk  that  the  Air  Force  can  induce  its  contractors  to  bear. 

Such  changes  result  from  a  basic  shift  in  relative  jower  that  increases  the  contractors’ 
responsiveness  and  reduces  the  Air  Force’s  cost  of  monitoring  their  contract  compliance. 

Other  aspects  of  contracting  are  at  least  as  important  as  contract  type  in  risk 
management.  The  choice  of  contract  type  affects  the  business  environment  for 
development;  contractors  clearly  prefer  a  cost-based  to  a  fixed-price  contract,  if  each  is 
offered  with  similar  terms.  But  a  contract’s  statement  of  work  and  special  clauses  can  be 
written  to  soften  the  risks  implied 
of  contract  type  by  their  effects  on 
monitor  contract  compliance  can  h 
terms  of  the  contract. 


by  a  fixed-price  contract,  potentially  dominating  the  choice 
contractor  risk.  And  the  techniques  that  the  SPO  uses  to 
ave  larger  effects  on  risk  management  than  the  formal 
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measured  cost. 

durability  and  damage  tolerance  assessment,  a  formal  risk  assessment 
method  used  to  evaluate  the  risks  associated  with  critical  parts  of  the 
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Defense  Acquisition  Regulations 

digital  electronic  engine  control,  a  key  component  of  the  FlOO-PW-220; 
enabled  the  improved  engine  to  avoid  stall-stagnations. 

Department  of  Defense 

Defense  System  Acquisition  Review  Committee,  predecessor  of  the  Defense 
Acquisition  Board  (DAB), 
engine  flight  hour 
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Force  could  uso  to  improve  aspects  of  engine  performance  not  allowed  by 
Congress  in  the  CIP, 

Engine  Monitoring  System 

Engine  structural  integrity  program,  an  engineering  approach  to 
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first  version  of  the  FlOO  used  to  power  the  F-15. 

first  version  of  the  FlOO  used  to  power  the  F-16. 
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a  GE-funded  demonstrator,  based  on  the  FlOl,  that  provided  the  basis  for 
the  FlOl  DFE. 

the  first  derivative  engine  that  GE  developed  for  the  F-15  and  F-16  and  a 
primar>'  focus  of  this  study. 

the  vanation  of  the  FllO  that  GE  developed  for  the  F-14. 

functional  configuration  audit,  a  critical  portion  of  the  product-verification 

process. 

firm-flxed-price,  a  contract  type  that  sets  price  independently  of  all  cost 
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full  flight  release,  the  second  step  in  the  four-step  development  process, 
flight  hours 
full-scale  development 
General  Accounting  Office 

General  Electric  Co.,  specifically  its  Aircraft  Engine  Business  Group  in 
this  Note. 

gear-type  fuel  pump,  a  key  component  in  improved  variations  of  the  FlOO, 
including  the  FlOO-PW-220. 

components  exposed  to  the  hot  gases  from  the  combustor,  including  the 
turbine  vanes  and  blades  and  combustor;  it  does  not  include  the  turbine 
shafts  or  disks. 

initial  flight  release,  the  first  step  of  the  four-step  development  process, 
increased-life  core,  a  key  component  in  improved  variations  of  the  FlOO, 
including  the  F100-PW-220. 

improved  performance  engines,  the  FlOO-PW-229  and  FllO-GE-129, 
derivative  engines  based  on  the  AFEs. 

independent  research  and  development,  a  form  of  contractor-directed  R&D 
funded  by  the  government  as  an  element  in  overhead  on  defense  contracts, 
initial  service  release,  the  third  step  of  the  four-step  development  process, 
knots-indicated  airspeed 

light-out  detector,  a  subcomponent  of  the  augmentor  included  in  improved 
versions  of  the  FlOO,  including  the  FlOO-PW-220. 
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limitation-of-govemment-obligation  clause,  a  key  clause  in  development 
contracts  that  are  not  fully  funded  to  limit  government  obligation  to  the 
level  of  funding  available, 
line  replaceable  unit 
main  fuel  pump 

maintenance  man  hours,  a  measure  of  maintenance  resources  required  to 
support  an  engine. 

mission  or  military  qualification  test,  a  test  used  to  clear  engines  for 
production  before  the  subject  developments  began, 
nonrecurring  investment  full-scale  development 

operational  capability  release,  the  last  step  in  the  four-step  development 
process. 

Office  of  the  Secretary  of  Defense 

Pratt  and  Wliitney,  a  subsidiary  of  United  Technologies  Corporation, 
program  businesa/technical  review,  a  bimonthly  review  in  the  Propulsion 
SPO  used  to  move  information  on  problems  and  alternative  solutions  up  to 
the  commtinder  and  across  the  SPO  matrix. 

preliminary  design  review,  a  review  of  plans  required  before  development 
begins. 

preliminary  flight  rating  test,  a  test  used  to  clear  an  engine  for  flight  test 
before  the  subject  developments  began. 

Prime  Item  Development  Specification,  a  criterion  for  completing  a 
development  task. 

research,  development,  test,  and  evaluation,  a  formal  category  of  federal 
funding. 

request  for  proposal 

Resolution  of  Operational  Problems  clause,  the  key  mechanism  for 
facilitating  flexibility  in  CIP  contracts. 

Secretau7  of  the  Air  Force 
significant-event  report 

specific  fuel  consumption,  a  measure  of  the  fuel  required  to  achieve  a 
certain  level  of  thrust. 

the  flight-weight  version  of  the  FI  10  developed  to  fit  in  the  F-15. 

System  Project  Office 
Tactical  Air  Command 

test-analyze-fix,  the  basic  approach  used  to  develop  engines. 

Pratt  and  Whitney  engine  used  to  power  the  A-7,  F-14,  and  F-111. 

General  Electric  engine  used  to  power  the  A-10  and  S-3. 

Rolls  Royce/Allison  engine  used  to  power  the  A-7. 

unplanned-event  report 

two-digit  symbol  for  the  Propulsion  SPO. 
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1.  INTRODUCTION 


Much  has  been  written  about  the  “Great  Engine  War”  between  General  Electric  and 
Pratt  and  Whitney  to  supply  engines  for  the  Air  Force’;;  F-15s  and  F-16s.’  Observers  from 
the  Department  of  Defense  (DoD)  and  industry  often  cite  this  production  competition  as 
evidence  of  the  potentially  strong,  positive  effect  of  competition  on  defense  contracting.  This 
case  also  provides  useful  information  on  development  procedures  for  new  weapon  systems. 
Much  less  has  been  said  about  the  development  of  the  two  Alternate  Fighter  Engines — Pratt 
and  Whitney’s  FlOO-PW-220  and  General  Electric’s  FllO-GE-100 — that  made  this  production 
competition  possible.  A  testament  to  the  success  of  these  developments  is  the  Air  Force’s 
decision  to  develop  a  follow-on  set  of  improved  performance  engines  using  a  similar 
competition. 

This  Note  examines  the  Alternate  Fighter  Engine  developments,  givd-ig  special 
attention  to  the  Air  Force's  methods  of  assessing  and  managing  risk  in  these  developments. 
By  risk,  we  mean  uncertaintj'  that  can  allow  negative  outcomes  during  a  development.  In 
our  analysis,  risk  increases  as  the  potential  negative  outcomes  increase  in  size  or  become 
more  likely. ^  Risk  is  inherent  in  any  weapon  development  program.  All  participants  in  the 
two  engine  developments  agree  that  the  assessment  and  management  of  such  risk  are 
central  to  the  management  of  an  engine  development. 

Some  risks  are  internal  to  a  development; 

•  The  managers'  expectations  about  the  technology  being  pursued  may  not  be 
realized,  with  good  or  bad  outcomes  relative  to  their  expectations. 

•  The  resources  and  time  required  to  make  progress  during  a  development  may 
differ  from  those  expected  by  the  managers. 

And  because  development  takes  time,  the  circumstances  surrounding  the  development  can 
change  before  the  development  is  complete: 

•  Changes  in  the  threat  or  other  political  changes  can  alter  the  nature  of  the 
system  expected  to  emerge  from  the  development. 


iSee  Drewes,  1987;  Kennedy,  1985;  Mayes,  1988;  Ogg,  1987. 
^Section  2  clarifies  this  definition. 
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•  Changes  in  the  high-level  support  for  the  effort  can  modify  the  funding  and  other 
resources  available  to  the  development. 

•  Technological  advance  elsewhere  may  have  unexpected  implications  for  any  one 
development  effort. 

•  Economic  changes  outside  the  development  may  unexpectedly  alter  the  cost  or 
availability  of  the  resources  required  to  conduct  the  development. 


These  changes  may  lead  to  management  decisions  to  change  the  final  performance  of  the 
system  or  the  time  and  resources  required  to  complete  development.^ 


TWO  ENGINE  DEVELOPMENT  PROGRAMS 

Table  1.1  summarizes  the  basic  physical  specifications  for  the  two  engines  examined  in 
this  Note.  Both  engines  have  similar  engineering  characteristics;  the  FllO  has  somewhat 
higher  thrust  and  weighs  more.  In  their  final  developed  forms,  they  could  both  be  used  in 
the  Air  Force  F-15  and  F-16  fighters.  A  slight  variation  on  the  FllO,  the  FllO-GE-400,  could 
power  the  Navy  F-14. 

In  addition  to  the  presence  of  competition  mentioned  above,  five  features  of  the  two 
engine  developments  may  have  special  relevance  to  acquisition  policy  in  the  future,  as 
defense  budgets  decline.  First,  interest  is  growing  in  “derivative”  developments— 
developments  that  build  directly  on  an  existing  design  and  do  not  attempt  to  take  a  major 

Table  1.1 

A  Comparison  of  the  FlOO-PW-220  and  FllO-GE-100 
Engine  Specifications 


Specification  FlOO-PW-220  FllO-GE-lOO 


Length,  in. 

210 

181.9 

Maximum  diameter,  in. 

47 

46.5 

Airflow,  Ib/sec 

228 

254 

Pressure  ratio 

24 

30.4 

Bypass  ratio 

0.63 

0.87 

Weight,  lb 

3218 

3895 

Thrust,  lb 

23,830 

27,000 

Specific  fuel  consumption 

0.73 

0.67 

SOURCES.  Ogg,  1987,  pp.  37,  39,  U  S.  Air  Force,  Aeronautical 
Systems  Division  Deputy  for  iSvpulsioD,  1986. 


3In  the  engine  community,  pcr/brmance  refers  primarily  to  the  thrust  or  thrust-to-weight  ratio 
of  an  engine.  In  the  broader  acquisition  community,  pcr/brmancc  includes  all  aspects  of  a  system's 
technical  characteristics.  For  engines,  performance  includes  thrust,  operability,  durability, 
supportability,  and  reliability.  Unless  otherwise  stated,  wc  use  the  broader  definition  in  this  Note. 


technological  step  into  the  unknown.  With  these  engines,  a  successful  program  of  derivative 
development,  the  Engine  Model  Derivative  Program  (EMDP),  was  initiated  in  the  Air  Force 
engine  community.  Second,  interest  is  growing  in  ways  to  maintain  flexibility  in 
development  programs  without  losing  oversight  or  control  of  the  programs.  These  engine 
programs  displayed  an  unusual  degree  of  flexibility;  more  generally,  all  Air  Force  engine 
programs  use  a  special  kind  of  development,  the  Component  Improvement  Program  (CIP), 
that  has  been  successful  in  maintaining  considerable  flexibility  without  losing  control. 

Third,  there  is  strong  policy  emphasis  today  in  DoD  to  move  away  from  the  use  of  fixed-price 
contracts  in  full-scale  development  and  toward  cost-based  contracts.  These  engine 
developments  used  an  extreme  form  of  fixed-price  contract,  a  firm-fixed-price  contract 
without  economic  price  adjustment,  that  operated  satisfactorily  for  the  Air  Force  and  its 
contractors.  Fourth,  interest  continues  in  the  development  of  improved  formal  risk 
assessment  and  risk  mcinagement  tools.  The  Air  Force  engine  community  developed  a  series 
of  such  tools  shortly  before  the  subject  engines  were  developed;  looking  at  these  tools’  effects 
on  these  engine  developments  provides  valuable  insights  into  their  potential  use  elsewhere. 
Fifth,  the  Propulsion  SPO  that  oversaw  these  developments  had  a  stable  structure  and  an 
experienced  and  expert  staff  in  place  before  these  developments  began,  and  it  was  beginning 
to  apply  a  consistent  doctrine  to  the  development  of  a  series  of  engines.  Although  that  SPO 
remains  in  place  today  and  has  overseen  a  series  of  successful  developments,  recent  events 
have  diminished  its  role  in  the  development  of  future  engines. 

APPROACH 

To  gather  information  on  the  two  engine  developments,  we  used  official  Air  Force 
documents,  secondary  sources,  and  interviews  with  key  Air  Force  and  contractor  personnel  in 
the  two  developments.  We  focused  on  the  1979-1986  period,  because  the  Air  Force  began  to 
fund  development  work  on  the  engines  in  1979  and  received  the  first  engines  from  those 
developments  in  the  field  in  operational  aircraft  in  1986. 

We  reviewed  relevant  contract  documents,  weekly  activity  reports,  and  biannual 
command  assessment  reviews  from  the  Propulsion  SPO;  Propulsion  SPO  briefings  on  the 
Alternate  F'ighter  Engine  competition  and  its  mejiagement  and  development  philosophy;  and 
official  Air  Force  histories  and  reports  to  the  Aeronautical  Systems  Division  History  Office 
from  the  Propulsion  and  related  aircraft  SPOs. 

We  used  this  material  to  prepare  for  interviews  on  the  technological,  political,  and 
managerial  environment  in  which  development  took  place,  and  on  major  events  during 
development  and  the  Air  Force  and  contractors’  methods  of  dealing  with  those  events.  These 
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interviews  ultimately  yielded  information  on  a  wide  variety  of  management  issues  related  to 
the  two  engine  developments.  Using  that  information  together  with  material  from  the 
documents  we  had  collected,  we  distilled  the  major  points  discussed  in  the  following  sections. 

Resource  constraints  prevented  us  from  covering  all  aspects  of  these  developments, 

We  focused  on  events  and  circumstances  within  the  developments  themselves,  especially 
within  the  SPO.  As  a  result,  we  focused  less  on  the  general  environments  in  the  Air  Force, 
General  Electric,  and  Pratt  and  Whitney  during  these  developments.  We  gave  very  little 
attention  to  the  markets  for  and  development  of  commercial  engines  related  to  the  engines 
addressed.  Future  analysis  should  addre  .^  tnese  topics. 

ROAD  MAP 

Section  2  briefly  reviews  our  view  of  risk  assessment  and  management  in  this  Note. 
Section  3  reviews  the  Air  Force’s  general  approach  to  engine  development  and  innovations  in 
risk  assessment  and  management  that  played  an  important  role  in  the  subject  developments. 
Section  4  offers  a  brief  history  of  the  two  developments  and  the  environment  in  which  they 
proceeded.  Section  5  examines  in  more  detail  the  risks  inherent  in  the  development.s  and 
three  key  factors  that  affected  the  management  of  risk  in  these  developments — competition, 
general  management  and  planning  practices,  and  contracts.  Section  6  concludes  the  study 
witli  simple  lessons  learned  and  implications  for  future  policy  and  policy  analysis. 
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2.  RISK  ASSESSMENT  AND  RISK  MANAGEMENT^ 


Weapon  system  development  is  aji  inherently  risky  activity — a  statement  with  which 
many  defense  personnel  and  contractors  would  agree  but  the  precise  meaning  of  which  would 
be  difficult  to  agree  on.  Most  would  concur  that  risky  connotes  that  weapon  system 
development  is  not  a  predictable  process,  and  that  the  activity  involves  many  surprises  or 
events  with  negative  outcomes.  That  is,  the  word  risk  suggests  not  only  unpredictability  but 
danger.  This  definition  becomes  especially  meaningful  when  we  discuss  not  just  risk  but  risk 
management.  Those  who  manage  risk  have  a  distinct  desire  to  ameliorate  the  negative 
effects  associated  with  the  unpredictability  of  a  weapon  system  development. 

If  we  accept  this  view,  many  ways  remain  to  define  precisely  what  risk  is,  The  very 
unpredictable  nature  of  risk  itself,  however,  tends  to  defy  further  formalization.  Any  attempt 
to  be  precise  about  what  risk  is  tends  to  sacrifice  some  aspect  of  unpredictability.  It  is 
hard — and  perhaps  even  misleading — to  characterize  too  precisely  a  situation  about  which 
we  are  profoundly  uncertain.  That  .said,  aneilysis  benefits  from  precision.  This  section  briefly 
examines  how  development  managers  view  risk,  risk  assessment,  and  risk  maunagement  and 
defines  these  concepts  to  order  our  inquiry  in  the  sections  that  follow.^ 

A  REALISTIC  WAY  TO  THINK  ABOUT  RISK  IN  ANALYSIS 

The  predominEmt  anedytic  definition  of  risk  is  probably  that  of  economists  and  decision 
theorists,  which  emphasizes  unpredictability.  For  economists,  risk  or  uncertainty  exists 
whenever  unpredictability  exists. ^  Risk  associated  with  a  process  increases  as  the  range  of 
possible  outcome.s  of  that  process  increases.  More  formally,  risk  increases  as  the  variEuice  of 
outcomes  associated  with  the  process  increases.  To  illustrate,  consider  the  two  distributions 
in  Figure  2.1.  The  outcome  of  a  process  is  represented  on  the  horizontal  axis  in  terms  of  a 
single  metric  of  performance.  Subjective  probability  density  lies  on  the  vertical  axis.  Based 
on  this  definition,  distribution  D1  is  riskier  than  distribution  D2  because  D1  is  more  diffuse 


slightly  revised  version  of  this  section  appears  as  Section  2  in  Camm,  1993. 

2After  this  work  was  complete,  Steven  Garber  of  RAND  brought  the  following  references  to  my 
attention.  They  confirm  that  the  views  of  risk  assessment  and  risk  management  that  we  observed  in 
the  Air  Force  are  consistent  with  those  observed  more  broadly  in  private  industry.  Cf  Garber, 
forthcoming;  MacCrimmon  and  Wehrung,  1986;  March  and  Shapira,  1987;  and  Shapira,  1986. 

^Many  economists  would  go  further  to  distinguish  risk  from  uncertainty.  Risk  occurs  when  the 
unpredictability  is  associated  with  the  outcomes  of  a  well- understood  stochastic  process;  uncertainty 
occurs  when  unpredictability  results  from  outcomes  of  a  poorly  understood  process.  A  related 
distinction  will  be  useful  to  us  below. 


-6- 


Figure  2.1 — Subjective  Probability  Density  Distributions  D1  and  D2  for  Two  Prog  ams 


than  D2.  Dl  is  riskier  even  though  the  central  tendency  for  D1  is  well  above  that  for  D2  and 
would  be  riskier  even  if  Dl  stochastically  strictly  dominated  D2.'‘ 

Now  suppose  that  Dl  and  D2  represent  the  anticipated  outcomes  of  two  different 
approaches  to  developing  a  weapon  system.  The  metric  of  performance  might  be  the  shop 
visitation  rate,  a  measure  of  supportability,  for  an  engine.  Viewing  these  alternatives, 
weapon  system  developers  would  agree  that  D2  represents  the  riskier  approach.  They  would 
justify  this  position  by  pointmg  out  that  poor  outcomes  are  more  likely  with  D2  than  with  Dl. 
Going  further,  some  might  be  willing  to  set  a  minimum  standard  measure  of  performance  S 
for  the  engine  and  characterize  risk  as  the  subjective  probability  associated  with  outcomes 
lower  than  this  standard.  For  example,  if  the  standard  were  S  in  Figure  2.2,  which  re¬ 
creates  the  distributions  in  Figure  2.1,  the  risk  associated  with  each  alternative  would  be 
proportional  to  the  shaded  areas  R1  and  R2,  representing  the  subjective  probabilities  that 
the  engine  designed  by  each  process  failed  to  meet  the  set  standard. 

Development  managers  would  find  this  view  of  their  decision  environment  grossly 
oversimplified.  For  example,  such  managers  do  not  generally  attempt  to  estimate,  even 

■•That  is,  suppose  that  we  imagine  random  draws  from  both  distributions  simultaneously.  If  we 
believe  that  outcomes  for  the  two  distributions  are  correlated  so  that  the  outcome  for  Dl  always 
dominates  that  for  D2,  then  Dl  stochastically  strictly  dominates  D2. 


s 

Metric  of  performance 

Figure  2.2 — Risks  Rl  and  R2  Associated  with  Two  Weapon  System  Development  Programs 
and  Occurring  Below  the  Set  Standard  for  Performance  S 

approximately,  the  probability  of  failure  as  defined  above  or  to  compare  such  estimates 
across  policy  alternatives.  Understanding  this,  we  can  still  use  the  approach  offered  in 
Figures  2.1  and  2.2  to  provide  a  useful  metaphor  for  thinking  about  decision  making  in 
weapon  system  development.  Let  us  continue  with  this  metaphor  for  a  moment  before 
returning  to  the  question  of  how  precise  managers’  views  of  risk  might  be. 

The  density  functions  in  Figures  2.1  and  2.2  are  essentially  risk  assessments.  Risk 
managers  ceuinot  effectively  make  such  assessments  independently  of  the  policies  they 
intend  to  use  to  manage  risk.  That  is,  they  effectively  view  risk  management  as  a  way  to 
alter  the  shape  of  the  distributions  shown.  At  any  point  during  a  development,  we  can  think 
of  the  mcuiager's  subjective  beliefs  about  the  program’s  outcome.  Such  beliefs  change 
through  the  course  of  a  development.  If  the  manager  expects  success  at  a  certain  point  in 
time,  he  or  she  has  adopted  policies  that  restrict  the  degree  of  risk  associated  with  such  areas 
as  Rl  and  R2  to  an  acceptably  low  level.  Some  of  those  policies,  such  as  an  acquisition  plan, 
system  specifications,  contract,  or  test  plan,  can  be  established  by  the  manager  today.  Some 
of  them  cannot  be  made  explicit  in  advance.  The  manager  must  expect  surprises,  the  details 
of  which  cannot  be  known  and  planned  for  in  advance.  Each  such  surprise  will  presumably 


alter  the  iWanager’s  risk  assessmont  and  force  a  change  in  policy  in  some  way  to  get  risk 
under  control  again. 

Viewed  in  this  way,  risk  management  begins  to  look  very  much  like  the  general 
management  of  n  development-  program.  And.  in  fact,  development  managers  draw  little 
distinction  between  the  two.  In  a  sense,  the  central  task  of  a  development  program  is  to 
eliminate  basic  uncertainties  about  a  new  design  so  that  it  can  be  transformed  into  a  useful 
product.  Doing  so  takes  time,  introducing  risks  associated  with  the  environment  in  wliich 
development  occurs  and  in  which  the  product  will  be  used.  Development  managers  are  quite 
comfortable  thinking  about  development  in  these  terms,  bringing  risk  management  per  se 
close  to  their  core  concerns  in  the  course  of  a  development. 

That  said,  risk  management — or  more  generally,  program  management— for  a 
development  is  much  more  complex  than  the  simple  metaphor  above  would  suggest. 
Managers  do  not  generally  think  in  terms  of  subjective  probability  densities  such  as  those 
presented  above.  They  think  more  in  terms  of  contingencies:  What  would  happen  if  this 
happened?  Roughly,  how  likely  is  it?  V.Tiat  kind  of  trouble  would  it  cause?  What  can  I  do 
now  to  mitigate  that  trouble?  What  kind  of  resources  or  staff  would  I  want  then  to  deal  with 
it?  This  process  of  assessing  risk,  planning  for  it,  and  reacting  to  it  is  what  we  want  to 
understand  better  in  this  Note.  The  metaphor  above  helps  us  understand  that  managers 
generally  make  such  assessments  by  focusing  on  surprises  that  can  hurt  them  and  seeking 
ways  to  mitigate  the  effects  of  those  surprises. 

A  SIMPLE  STRUCTURE  FOR  INQUIRY 

Surprises  come  from  many  sources.  They  affect  a  development  program  in  a  variety  of 
wavs.  And  managers  have  a  number  of  tools  for  planning  for  and  responding  lo  surprises. 

Sources  of  Risk 

Managers  look  for  surprises  in  two  places:  outside  the  development  and  within  it. 
First,  development  takes  time.  W»'ile  it  occurs,  the  world  outside  a  development  can 
change,  precipitating  surprises  for  a  development  program.  Most  basically,  changes  in  the 
threat  can  affect  either  willingness  to  continue  funding  the  program  or  the  requirements  set 
for  the  final  product.  Changes  in  technology  can  affect  the  availability  of  subsystem 
capabilities  relied  on  by  the  development  or  the  need  for  the  system  under  development. 
Changes  in  the  economy  can  modify-  the  cost  of  the  development  itself,  that  of  the  final 
product,  or  the  availability  c  f  funds  to  maintain  the  development  as  expected.  Changes  in 
the  Air  Force  testing-and-evaluation  community  can  affect  the  availability  oi  test  assets.  All 
these  factors  are  essentially  beyond  the  manager's  control.  However,  their  effects  can 
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generally  be  reduced  by  restricting  the  length  of  a  development,  so  that  fewer  opportunities 
for  surprises  arise  over  the  course  of  the  development.  More  likely,  the  manager  must 
anticipate  specific  types  of  surprises  and  tailor  individual  responses  to  each  type. 

Second,  even  if  the  world  outside  the  development  remains  stable,  surprises  can  be 
expected  from  within  the  development.  Examples  are  development  efforts  that  require  more 
time  or  resources  than  expected  to  reach  a  particular  performance  improvement,  and  certain 
technical  goals  set  m  the  program  that  turn  out  to  be  infeasible.  The  manager  has  greater 
control  over  such  factors,  but  can  stiU  not  expect  to  eliminate  such  surprises. 

Program  Attributes  Affected  by  Risk 

When  surprises  occur,  they  can  affect  a  number  of  program  attributes.  First  and 
foremost,  they  can  affect  the  probability  that  the  program  will  survive  to  316^  a  useful 
product.  Assuming  successful  program  completion,  they  can  aifect  the  resources  and  time 
required  to  complete  the  program;  these  are  the  “cost"  and  “schedule”  criteria  normally 
associated  with  development.  Suprises  can  also  affect  various  measures  of  final  system 
“performance.”  Traditional  measures  of  system  performance  emphasize  combat  capability 
and  can  normally  be  measured  in  a  variety  of  ways  specific  to  each  system.  Logistics- 
oriented  factors  such  as  reliability,  availability,  maintaindbilit>’,  and  operating  and  support 
costs  are  increasingly  considered  important  parts  of  system  performance.  Producibility  and 
production  cost  for  the  system  round  out  the  performance  factors  relevant  to  the  manager. 

As  a  development  program  is  normally  defined,  a  manager  will  have  a  hard  time 
meeting  his  or  her  goals  on  all  of  the  above  factors.  To  increase  the  probability  of  program 
survival  early  in  the  progrsun’s  life,  the  manager  must  make  the  program  look  attractive 
relative  to  alternative  programs.  Hence,  the  manager  generally  attempts  to  understate  goals 
for  development  cost  and  schedule  and  overstate  the  performance  goals  of  the  system.  To  the 
extent  that  such  goals  are  adopted  as  standards  like  those  in  Figures  2.1  and  2.2 — that  is,  a 
program  fails  if  it  fails  to  meet  all  its  goals — misstatement  of  goals  actually  increases  the  risk 
associated  with  a  program.  In  most  cases,  however,  the  manager  must  accept  such  risk  to 
reduce  the  risk  of  losing  overall  support  for  the  program  to  a  competing  development 
program.  Managers  well  understand  this  tension  between  the  goals  cf  program  survival  and 
other  goals  of  the  program;  they  accept  it  essentially  as  a  price  of  entry  for  conducting 
development  activities.  In  the  end,  however,  such  acceptance  means  that  the  manager 
cannot  expect  to  meet  his  or  her  goals  and  must  expect  to  make  trade-offs  in  allocating 
shortfalls  among  goals. 
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When  surprises  occur,  the  manager  must  again  make  trade-offs  among  goals.  Some 
surprises  will  loosen  constraints  on  the  manager;  an  unexpectedly  high  performance  outcome 
ir.  one  area  may  allow  the  manager  to  reduce  risk  associated  with  performance  in  another 
area  or  to  hold  the  line  on  the  costs  or  schedule  of  development.  Negative  surprises,  on  the 
other  hand,  will  lead  a  manager  to  spread  the  negative  effects  across  goals.  A  tost  failure,  for 
example,  may  lead  to  a  schedule  slip  and  additional  development  work  to  achieve  the  initial 
performance  goal  at  the  expense  of  schedule  and  cost  goals. 

How  a  manager  makes  such  trade-offs  should  depend  on  the  relative  priorities  that  he 
or  she  places  on  different  goals,  based  either  on  guidance  from  higher  echelons  of  government 
overseeing  the  project  or  his  or  her  own  personal  goals.  We  should  expect  these  priorities  to 
differ  from  one  development  program  to  another  and  perhaps  even  to  change  over  the  course 
of  a  development.  Patterns  in  such  trade-offs  are  of  great  interest  to  us. 

Methods  for  Anticipating  and  Responding  to  Risk 

A  manager  can  use  two  basic  approaches  to  plan  for  or  react  to  surprises.  The  first 
emphasizes  formal  documents  and  processes.  The  second  approach  emphasizes  good  people. 

The  first  approach  uses  the  performance  specifications  for  a  new  system  to  set  the 
general  level  of  risk  for  the  program:  more  ambitious  specifications  are  riskier.  The  approach 
spells  out  a  formal  acquisition  plan  for  the  development,  specifying  lines  of  authority,  the 
nature  of  competition,  or  prototyping  used  during  the  development.  It  uses  contracts  and 
memoranda  of  understanding  and  agreement  to  balance  the  concerns  of  the  Air  Force  and 
other  parties  to  the  development.  It  uses  a  master  test  plan  to  anticipate  required  testing 
assets,  set  sequences  of  events,  and  respond  to  test  failures  over  the  course  of  the 
development.^ 

Broadly  riewed,  a  development  is  a  test  program  that  repeatedly  tests  newly 
developed  systems,  analyzes  problems  identified  during  test,  and  fixes  them  in  preparation 
for  another  test.  This  “test-analyze-fix”  or  TAF,  approach  applies  at  the  macro  level  as  a 
metaphor  for  the  program  as  a  whole  and  can  be  applied  in  a  mere  targeted,  explicit  way  to 
deal  with  specifically  identified  problems. 

Formal  risk  assessment  can  be  associated  with  any  one  of  these  activities.  Foi*mal  risk 
assessment  works  best  when  the  processes  in  question  are  well  understood  cmd  good  data 

SSuch  documents  create  a  paper  trail  that  developers  can  use  to  cover  themselves  if  things  go 
wTong;  that  is,  they  can  prove  that  they  did  everything  that  was  required  of  them,  We  are  more 
interested  in  the  way  developers  use  such  docun;ents  to  anticipate  risk  and  plan  for  it;  that  is,  we  are 
interested  in  how  a  creative  planning  process  can  lailcr  documents  to  a  development  program’s  needs, 
not  simply  fulfill  regulatory  requirements. 
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exist  on  those  processes.  Hence,  it  is  most  likely  to  support  design  of  selected  parts  of  the 
test  program  or  of  warranties  included  in  a  contract.  Risks  associated  with  the  development 
as  a  whole,  as  noted  above,  are  more  difficult  to  state  in  clear,  quantitative  terms.  Formal 
risk  assessment  is  of  limited  use  in  such  a  sotting.^ 

The  presence  of  contracts  among  these  tools  raises  an  important  point  about  risk. 

Risk  can  be  perceived  from  different  perspectives.  For  example,  although  a  contractor  is 
probably  better  able  than  the  Air  Force  to  affect  surprises  that  arise  in  the  day-to-day 
development  of  a  system,  the  Air  Force  is,  presumably,  better  able  than  the  contractor  to 
affect  overall  funding  for  a  development  program.  Contracts  can  be  written  to  shift  the 
effects  of  surprises  toward  those  parties  best  able  to  mitigate  their  effects.  Hence,  the 
contractor  often  bears  much  of  the  risk  associated  with  unexpected  cost  growth  during  a 
development,  whereas  the  government  bears  the  risk  associated  with  premature  termination 
of  a  program.  More  generally,  the  Air  Force  as  a  whole  is  probably  better  able  to  bear  the 
effects  of  surprises  than  is  an  individual  contractor.  Contracts  can  be  written  to  shift  the 
effects  of  surprises  that  cannot  be  mitigated  to  the  party  better  able  to  bear  such  effects.  In 
practice,  of  course,  even  when  the  Air  Force  as  a  whole  can  bear  large  negative  outcomes, 
officers  in  a  SPO  concerned  about  their  futures  in  the  Air  Force  probably  cannot.  Hence, 

SPO  managers  may  resist  bearing  risks  best  borne  by  the  Air  Force. 

Once  a  contract  is  negotiated,  it  splits  the  effects  of  many  surprises  so  that  one  party 
benefits  from  the  surprise  while  the  other  is  injured.  Our  approach  to  risk  implies  that  one 
party  need  not  associate  any  risk  with  a  surprise  that  imposes  a  substantial  risk  on  the  other 
party.  For  example,  under  a  fixed-price  contract,  the  contractor  bears  the  full  risk  of 
unexpected  cost  increases  while  the  Air  Force  feels  no  effect.  On  the  other  hand,  the 
contractor  enjoys  the  full  benefits  of  unexpectedly  low  costs.  When  costs  are  unexpectedly 
low,  the  Air  Force  can  see  such  a  benefit  as  a  foregone  benefit  for  the  Air  Force — that  is,  as  a 
negative  outcome.  That  is,  dlthough  the  fixed-price  contract  determines  their  costs.  Air  Force 
officials  may  view  the  foregone  benefit  as  a  risk  worth  planning  against.  Such  a  perception 
complicates  our  simple  view  above  that  risk  is  associated  only  with  negative  outcomes. 

Although  formal  contracts  are  written  only  between  the  .\ir  Force  and  contractors  or 
between  contractors,  these  considerations  apply  to  many  other  situations  in  which  more  than 
one  party  plays  a  role,  A  SPO  typically  has  many  relationships  with  other  SPOs,  test 
facilities,  other  parts  of  the  Air  Force,  and  sometimes  other  services.  Some  of  these 
relationships  are  codified  in  memoranda  of  understanding  and  agreement;  others  rely  on 


®For  a  farther  discussion  of  these  points,  see  Bodilly,  Camm,  and  Pei,  1991. 
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established  custom.  In  all  cases,  more  than  one  perspective  on  the  risks  associated  with  a 
particular  surprise  is  possible  and  can  affect  how  managers  plan  for  and  react  to  that 
surprise. 

The  second  basic  approach  to  planning  for  and  reacting  to  risks  is  quite  different  from 
that  taken  above.  It  relies  on  good  people  rather  than  on  documents  and  procedures.  At 
some  level,  good  people  are  required  to  negotiate  and  prepare  documents  and  procedures. 
The  point  here  is  that  a  good  staff  adds  value  beyond  the  planning  function.  In  fact,  a  good 
staffs  primary  value  may  well  lie  not  in  planning  for  the  futui  e  bu>.  in  reacting  confidently 
and  creatively  when  things  go  wrong. 

The  importance  of  good  people  is  repeatedly  emphasized  by  development  managers. 
Contractors  favor  coveted  development  programs  not  just  with  resources,  but  with  their  best 
people.  The  Air  Force  responds  by  allocating  its  best  people  to  its  highest  priority 
development  projects.  Such  practices  occur  in  production  programs  as  well.  They  take  on  a 
special  meaning  in  development  programs  because  of  the  nature  of  the  risks  present  in  those 
programs. 

Although  good  planning  can  provide  a  framework  for  dealing  with  routine  risks — risks 
encountered  in  the  past  or  risks  that  are  fairly  obrious  in  a  new  program— -it  cannot  manage 
well  the  totally  unexpected.  When  the  totally  unexpected  occurs,  well-inibrmed  and  timely 
discretion  is  required  to  respond  to  the  surmse.  The  better  the  staff  available  to  do  this  is, 
the  better  is  the  response  and  the  less  managers  must  rely  on  the  blunter  rules  that  an 
acquisition  plan  or  contract  might  incorporate  to  manage  surprise.  A  we’l-organized, 
competent  staff  offers  an  additional  benefit  in  the  face  of  uncertainty,  Because  surprises  can 
bring  benefits  as  well  as  risks,  the  presence  of  a  solid  staff  allows  mar  agers  to  maintain 
greater  flexibility  in  a  program  to  exploit  opportunities  as  they  arise. 

The  presence  of  parties  with  different  points  of  view,  of  course,  complicates  the  use  of 
skilled  people  to  respond  flexibly  to  surprises.  Each  surprise  offers  opportunities  to  reopen 
an  agreement  made  earlier  to  change  the  balance  achieved  earlier.  Among  the  skills  in  a 
well-organized  staff  will  typically  be  abilities  to  exploit  such  opportunities.  However, 
exploiting  surprise  to  renegotiate  earlier  agreements  can  damage  the  basic  relationship 
between  two  parties  over  the  long  term  if  it  happens  repeatedly,  ultimately  leading  to  more 
rigid  Ekrraugements  designed  to  discourage  such  exploitation,  even  if  they  stifle  the  flexibility 
that  allows  a  program  to  benefit  from  pleasant  surprises.  Such  exploitation  is  most  likely  to 
occur  when  the  skills  of  two  parties  are  not  well  balanced.  For  example,  if  an  inexperienced 
SPO  faces  a  contractor  using  a  team  with  considerable  experience  working  together,  we  can 
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expect  trouble  downstream  cs  surprises  provide  opportunities  for  the  contractor  to  exploit  its 
greater  experience. 

SUMMARY 

Although  our  primary  interest  lies  in  risk  assessment  and  risk  management  in 
development  programs,  the  nature  of  development  activities  suggests  that  we  should  be 
prepared  to  examine  a  fairly  broad  range  of  management  activities.  This  is  true  despite  our 
narrow  definition  of  risk  as  the  presence  of  unpredictable  events  with  negative  consequences. 
Many  development  managers  use  this  definition  and  see  their  task  as  general  managers  as 
one  of  identifying  and  controlling  such  risk  over  the  course  of  a  development. 

Surprises  can  arise  outside  a  development  program — beyond  the  control  of  those 
involved  in  the  development — or  much  closer  to  home  and  closer  to  their  control.  They  can 
affect  the  basic  survival  of  a  program  or,  assuming  that  it  survives,  the  cost,  schedule,  and 
performance  associated  with  the  program.  Development  managers  use  two  different 
approaches  to  plan  for  and  react  to  surprises.  They  develop  documents  and  procedures  that 
define  both  risk  and  the  methods  for  reducing  or  redistributing  it.  These  include  system 
specifications,  acquisition  plans,  contracts,  memoranda  of  understanding  and  agreement,  and 
test  plans.  And  development  n^anagers  develop  and  nurture  experienced,  skilled  staffs. 
Without  such  staffs,  managers  cannot  respond  adequately  to  the  wide  range  of  surprises  that 
arise.  With  them,  they  can  rely  less  on  formal  documents  and  processes  and  thereby 
maintain  the  flexibility  that  allows  them  to  take  advantage  of  new  opportunities  as  they 
arise. 

In  the  end,  then,  it  is  difficult  to  distinguish  risk  management  from  general 
management.  In  the  following  sections,  we  explore  a  broad  range  of  management  issues. 
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3.  THE  AIR  FORCE’S  GENERAL  APPROACH  TO  ENGINE  DEVELOPMENT 


In  the  late  1970s,  the  Air  Force  was  actively  changing  its  entire  approach  to  the 
development  of  engines.  A  series  of  unsatisfactory  development  programs  led  the  Air  Force 
to  believe  that  it  needed  more  continuity  in  engine  development  to  make  sure  that  new 
developments  could  learn  more  satisfactorily  lessons  from  the  past.  The  Air  Force  also 
sought  a  more  sophisticated  approach  to  nsk  assessment  that  would  improve  its  ability  to 
predict  sources  of  failures  in  engines  and  use  this  information  to  improve  engine  operability, 
supportability,  and  durability.  In  particular,  it  sought  better  methods  to  predict  the  behavior 
of  on  engine  over  its  lifetime  and  use  this  information  to  extend  its  lifetime.  To  achieve 
continuity  in  engine  development,  the  Air  Force  organized  a  new  SPO  in  1977  to  oversee  all 
engine  system  development.  The  Propulsion  SPO  became  the  focal  point  for  consolidating 
and  applying  a  rapidly  growing  body  of  new  development  concepts  that  the  Air  Force  hoped 
would  help  it  improve  the  supportability  and  durability  of  new  engines. 

This  is  the  environment  in  which  the  development  programs  for  the  FlOO-PW-220  and 
FllO-GE-100  engines  began.  Among  the  first  Air  Force  engines  to  benefit  from  this  new 
approach,  they  began  development  too  soon  to  benefit  from  the  full  application  of  the  new 
concepts,  but  they  offer  a  good  example  of  initial  applications  that  helped  shape  the  Air 
Force's  continuing  effort  to  improve  its  engine  development  methods.  This  section  opens 
with  a  brief  discussion  of  the  way  the  Propulsion  SPO  thinks  about  risk,  risk  assessment, 
and  risk  management,  and  how  its  approach  to  risk  management  affects  the  broader 
management  of  engine  development.  It  then  gives  a  brief  overview  of  the  development 
concepts  the  Air  Force  and  its  contractors  were  instituting  in  the  late  1970s  to  help  assess 
and  manage  risk,  paying  special  attention  to  the  accelerated  mission  test,  the  four-step 
development  process,  and  the  Engine  Structural  Integrity  Program  (ENSIP).  Sections  4  and 
5  say  more  about  the  effects  of  these  new  methods  on  the  FlOO-PW-220  and  FllO-GE-100 
developments. 

MANAGING  RISK  IN  THE  ENGINE  DEVELOPMENT  PROCESS 

The  management  of  engine  development  is  risk  management.  This  is  the 
overwhelming  consensus  of  managers  in  the  Propulsion  SPO  and  engine  contractors.  System 
development  by  its  very  nature  involves  the  systematic  identification,  reduction,  and 
management  of  risks.  And  as  complex,  technologically  sophisticated  systems  whose 
performance  is  critical  to  personnel  safety,  the  survivability  of  other  Air  Force  systems  in  an 
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airframe,  and  mission  success,  engines  present  an  array  of  development  risks  that  has  forced 
engine  developers  to  learn  to  assess  and  manage  such  risk.  As  important  as  this 
technological  risk  is,  it  must  be  understood  in  a  broader  context.  To  create  and  maintain 
support  for  an  engine  development  process,  the  Air  Force  and  its  contractors  must  also 
manage  a  broad  range  of  political  and  funding  risks.  The  Propulsion  SPO  was  created  in 
part  to  enable  the  Air  Force  to  better  understand  and  manage  these  risks  in  engine 
development.  It  is  proud  of  the  success  it  has  had  in  doing  so  and,  in  particular,  of  the 
success  it  had  in  managing  the  FlOO-PW-220  and  FllO-GE-100  developments  early  in  its 
existence. 

Two  kinds  of  risk  management  are  important  in  engine  system  development — 
planning  for  future  surprises  and  managing  surprises  that  actually  occur.  Planning  for  the 
future  considers  surprises  that  can  occur  throughout  an  engine's  life  cycle; 

•  design 

•  manufacture 

•  operation 

•  maintenance. 

The  Propulsion  SPO  focuses  on  the  first  two  stages  but  influences  the  others  through  the 
pilot’s  handbook,  logistics  planning,  and  other  products  that  it  manages  as  part  of  the 
development  process.' 

Planning  during  the  design  stage  seeks  a  low-risk  design  and  then  seeks  a 
management  plan  that  minimizes  the  risk  of  the  activities  required  to  transform  the  design 
into  a  final  configuration  ready  for  production.  Actual  development  uses  a  pragmatic  test- 
analyze-fix  process  that  iteratively  tests  a  particular  engine  configuration,  analyzes  failures 
in  that  configuration,  and  posits  solutions  to  be  tested  in  a  revised  configuration.  This 
process  continues  until  the  configuration  stabiUzes,  as  it  ultimately  must  for  the 
development  process  to  succeed. 

Planning  during  the  manufacturing  stage  “perfects”  the  final  product  design  by 
identifying  variabilities  in  the  actual  materials  used  in  manufacturing,  the  skills  of 
manufacturing  labor,  and  the  engineering  drawings  used  to  specify  the  manufacturing  design 
of  parts,  'iuch  variabilities  can  all  lead  to  quality  assurance  problems.  Once  it  identifies 

iDevelopers  of  other  kinds  of  systems  would  obviously  face  similar  types  of  general  risk- 
management  concerns.  How  they  dealt  with  them,  however,  would  depend  on  the  specific  problems 
faced  in  a  particular  development. 
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these  variabilities,  the  Propulsion  SPO  and  its  contractors  must  manage  them.  Management 
of  such  manufacturing- related  risks  has  changed  substantially  over  the  last  ten  years; 

•  Manufacturing  has  become  more  integrated  with  design,  in  part  because 
problems  with  affordability  and  safety  discovered  in  the  1970s  could  only  be 
addressed  by  solving  manufacturing  problems. 

•  Technology  allows  better  analysis  and  quality  assurance — for  example,  the 
Propulsion  SPO  can  now  x-ray  new  turbine  blades  before  accepting  them. 

•  Despite  contractor  resistance,  the  Propulsion  SPO  uses  structured  manufacturing 
reviews  to  monitor  what  manufacturers  have  done, 

In  such  planning,  developers  must  weigh  these  technical  risks  against  other  risks.  The  most 
important  risks  are  associated  with  initiating  and  maintaining  general  support  for  a 
development  program.  The  continuing  competition  for  funds  can  easily  lead  developers  to 
overpromise  in  terms  of  final  system  performance  or  the  cost  or  schedule  of  development. 
Over-optimistic  promises  increase  the  risks  of  conducting  a  development  by  forcing  the 
developer  to  run  fewer  tests,  take  less  time,  or  work  against  less  realistic  performance  goals 
than  would  a  development  concerned  only  with  technical  risk.  Many  managers  agree  that 
the  principles  for  controlling  risk  in  a  development  are  well  understood  and  not 
controversial — current  DoD  documents  state  them  clearly.  But  they  are  difficult  to  apply  in 
the  broader  environment  that  sets  the  requirements  for  and  funds  system  developments. 

When  a  surprise  occurs,  the  Propulsion  SPO  must  first  manage  the  peuticular  event 
itself,  then  figure  out  its  implications  for  the  fleet,  and  finally  act.  Detcnnining  implications 
for  the  fleet  is  not  a  serious  concern  during  development,  because  there  is  little  at  risk.  The 
key  goal  here,  however,  is  not  to  allow  a  failure  or  an  error  corrected  in  one  development 
program  to  reoccur  in  another.  That  principle  calls  for  some  coordination  across  development 
programs,  even  if  it  does  not  have  as  high  a  priority  as  monitoring  effects  importemt  to  the 
tleet  as  a  whole. 

The  Propulsion  SPO  and  its  contractors  believe  that  two  general  factors  have  given 
them  an  exceptional  capability  in  risk  assessment  and  management: 

•  Failure  in  an  engine  can  have  catastrophic  consequences,  for  the  pilot  and  for 
other  subsystems  on  the  aircraft.  Knowing  this  has  made  engine  developer:  well 
attuned  to  the  need  for  careful  risk  assessment  and  has  led  them  to  adopt  new 
techniques  as  they  have  become  available. 
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•  The  Air  Force  is  now  working  with  the  fifth  generation  of  military  engines.  That 
experience  has  allowed  the  Propulsion  SPO  and  its  contractors  to  accumulate  a 
great  deal  of  practical  knowledge  about  potential  failures  and  methods  of  dealing 
with  them.  They  have  a  substantial  technical  base  to  build  on  and  experienced 
personnel  to  exploit  it.  When  the  Air  Force  and  its  contractors  face  new 
uncertainties,  they  have  established  methods  for  dealing  with  those  uncertainties 
and  experienced  personnel  familiar  with  these  methods. 

The  Air  Force  and  its  contractors  had  less  experience  when  they  developed  the  FllO-GE-100 
and  FlOO-PW-220.  But  they  were  attuned  to  the  problem  of  risk  assessment  and  working 
hard  during  those  developments  to  apply  what  they  knew.  The  remainder  of  this  section 
addresses  a  set  of  formal  concepts  and  methods  that  they  were  refining  during  the 
developments.  These  formal  methods  played  an  important  part  in  assessing  and  managing 
the  technical  risks  faced  by  the  Air  Force  and  its  contractors  during  the  two  developments. 

But  engine  developers  all  stress  that  risk  management  involves  substantially  more 
than  formal  methods.  By  its  very  nature,  risk  deals  with  unpredictable  circumstances, 
circumstances  that  do  not  comfortably  fit  in  any  preconceived  and  well-defined,  formal 
framework.  In  the  end,  the  mosc  important  risk  management  occurs  in  the  judgments  of 
experienced  development  managers.  One  of  the  most  important  legacies  of  the  developments 
we  examine  here  has  been  their  contribution  to  the  education  of  a  set  of  managers,  many  of 
whom  have  worked  together  in  the  Propulsion  SPO  for  over  a  decade.  They  have  been  able  to 
carry  their  accumulated  experience  in  managing  risk  from  one  engine  development  to  the 
next.  We  should  not  lose  sight  of  the  fact  that  the  management  of  system  development  is 
risk  management  as  we  review  important  but  nturower  improvements  in  techniques  of 
technical  risk  assessment  and  management.  We  review  in  turn  the  accelerated  mission  test, 
the  four-step  engine  development  process,  and  the  engine  structural  integrity  program 
(ENSIP). 

THE  ACCELERATED  MISSION  TEST 

The  accelerated  mission  test  (AMT)  runs  a  ground-test  engine  for  hundreds  or 
thousands  of  hours  to  simulate  the  stresses  it  would  experience  in  flight  over  the  course  of  an 
actual  lifetime  in  a  military  aircraft.  Now  used  routinely  in  engine  development,  the  .AMT 
marked  a  dramatic  departure  from  earlier  development  practices  in  the  1970s. ^ 


2\Ve  did  not  have  time  to  contrast  these  changes  with  those  occurring  in  commercial  engine 
development  during  the  same  period.  Although  the  major  engine  developers  build  both  commercial  and 
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In  engine  development  in  the  19608  and  early  19708,  little  attention  was  given  to  the 
specific  way  an  engine  would  be  used  in  an  aircraft.  Emphasized  instead  were  performance 
characteristics,  such  as  thrust,  rather  than  the  durability  of  engines  or  factors  that  would 
affect  their  long-term  operating  and  support  costs.  In  fact,  the  Air  Force  had  great  difficulty 
predicting  the  durability,  operability,  or  supportability  of  new  engines  under  development. 

During  that  period,  the  Air  Force  used  a  50-hr  preliminary  flight  rating  test  (F'FRT)  to 
qualify  engines  for  flight  test  and  a  150-hr  military  or  mission  qualification  test  (MQT)  to 
qualify  them  for  production.  Such  evaluations  had  two  basic  problems. 

First,  such  short  tests,  even  if  configured  to  reflect  the  type  of  activity  expected  when 
the  engine  flew  in  an  operational  aircraft,  could  not  simulate  the  stress  that  the  engine 
would  experience  over  its  full  lifetime — its  period  in  service  between  depot  overhauls,  They 
would  not  even  help  the  designer  determine  what  that  lifetime  would  be,  for  the  engine  or 
critical  constituent  parts  of  it.  The  tests  considered  only  high-power,  high-temperature 
conditions  and  sought  to  identify  (and  hence  allow  designers  to  eliminate)  high-cycle  fatigue 
failure  modes.  They  could  not  identify  low-cycle  fatigue  failure  modes  driven  by  extended  use 
in  the  field. 

Second,  the  tests  did  not  accurately  reflect  the  types  of  missions  anticipated  for  an 
engine  during  its  operation.  As  a  result,  even  if  the  tests  had  been  extended  to  allow 
exploration  of  low-cycle-fatigue  failure  modes,  they  would  not  stress  the  engine  parts  as  they 
would  be  stressed  over  their  lifetimes  during  operation.  That  is,  the  tests  could  not  identify 
the  specific  failure  modes  likely  to  occur  during  operation  and  hence  the  designs  and 
maintenance  practices  required  to  ameliorate  and  manage  those  failure  modes. 

The  AIVIT  overcame  both  problems  by  simulating  the  full,  expected  life  of  the  engine. 
Simulation  begins  with  a  “duty  cycle”  that  describes  as  accurately  as  possible  the  engine's 
expected  pattern  of  operational  usage  over  a  composite  mission.  The  duty-cycle  analysis 
begins  by  identifying  missions  and  identifying  the  types  and  durations  of  all  engine¬ 
damaging  events  associated  with  each  mission.  For  example,  air-to-air,  air-to-ground,  and 
navigation  missions  each  generate  predictable  patterns  of  speed,  altitude,  and  engine  throttle 
setting.  As  the  importance  of  these  patterns  became  clear  in  the  1970s.  the  Air  Force  began 
a  systematic  collection  of  data  on  patterns  experienced  in  actual  flight  operations.  These 
data  can  be  translated  into  specific  series  of  throttle-dwell  times  at  idle,  cruise,  intermediate, 

military  engines  and  General  Electric  in  particular  has  attempted  to  use  similar  engines  in  both 
markets,  the  innovations  described  below  respond  to  distinctly  military  problems  associated  with  the 
high  stress  placed  on  a  high-performance  fighter  engine. 
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and  max  afterburner  speeds,  as  well  as  afterburner  lights.  A  duty  cycle  is  simply  a  composite 
of  these  series.  Figure  3.1  provides  an  example. 

Repeating  this  cycle  to  reflect  the  total  number  of  these  composite  missions  in  the 
engine’s  lifetime  then  generates  the  engine's  complete  usage  profile.  The  developer 
compresses  this  profile  by  removing  non-damaging  steady-state  cruise  time  and  retaining 
damaging  cycles  and  hot  time.  He  adds  service  activities,  including  scheduled  and  on- 
condition  maintenance,  that  would  occur  in  the  field.  The  final  product  is  a  simulation  of  the 
engine’s  lifetime. 

The  AMT  runs  an  engine  through  this  simulation,  stopping  to  replace  damaged  parts 
and  periodically  to  inspect  the  condition  of  the  core  or  the  engine  as  a  whole.  When  the  AMT 
is  completed,  the  engine  and  its  components  have  effectively  experienced  a  full  lifetime  of 
flight  stress.  Such  a  test  allows  developers  to  assess  the  accuracy  of  their  predictions  for 
specific  components,  judge  the  actual  durability  of  those  components,  and  design  and  test 
alternative  component  designs  and  maintenance  plans  to  sustain  flight  safety  and  engine 
durability.  Designers  in  fact  expect  different  engine  parts  to  have  different  lifetimes.  An 
AMT  can  be  performed  for  each  part  of  the  engine  to  reflect  the  lifetime  expected.  For 


•  AMT  eliminates  nondamaging  testing 


SOURCE:  RadloH.  1985 


Time  (hr) 


Figure  3.1 — Illustrative  Engine  Duty  Cycle 
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example,  the  engines  we  examine  achieved  a  4,000-cycle  hfe  in  their  hot  sections — 
components  exposed  to  the  hot  gases  from  the  combustor— and  an  8,000-cycle  life  in  their 
cold  sections — the  remaining  components.^ 

Engine  testing  is  costly  and  time-consuming.  But,  without  empirical  tests,  analytic 
methods  of  predicting  the  effects  of  stress  on  engine  structure  are  not  good  enough  to  rely  on. 
The  compression  used  in  the  AMT  is  designed  to  reduce  the  cost  and  time  associated  with 
such  testing  so  that  the  AMT  can  be  used  iteratively  in  a  design  effort  that  repeatedly  tests 
components  or  maintenance  concepts  and  then  introduces  potential  improvements  for  further 
testing.  The  Air  Force  and  its  contractors  developed  and  refined  this  technique  in  the  1970s 
on  the  FlOO,  TF34,  TF41,  and  FlOl.  Using  AMT  to  allow  design  improvements  early  in  the 
life  of  the  TF34,  for  example,  significantly  cut  Component  Improvement  Program  (CIP) 
expenditures  for  major  design  upgrades  relative  to  those  on  the  TF41.'*  The  AMT  had 
become  a  standard  part  of  the  development  process  by  initiation  of  the  programs  we  are 
stud>'ing. 

FOUR-STEP  DEVELOPMENT  PROCESS 

During  the  1960s  and  early  1970s,  the  Air  Force  used  a  two-step  development  process 
in  full-scale  development  based  on  the  PFRT  and  MQT.  An  engine  was  first  qualified  for 
flight  testing  and  then  qualified  for  production.  By  the  late  1970s,  the  Air  Force  had  moved 
to  a  four-step  process  that  was  more  a  conceptual  framework  for  structuring  a  full-scale 
development  than  a  formal  regulatory  structure  that  had  to  be  obeyed  in  each  development.® 
Table  3.1  matches  the  steps  of  this  process  to  steps  in  the  development  process  for  an 
airframe  and  to  steps  in  the  earlier  engine  development  process. 

Roughly  speaking,  the  first  step,  initial  flight  release  (IFR),  emphasized  flight  safety 
so  that  developers  could  fly  the  aircraft  enough  to  learn  about  the  engine.  Flight  was 
expected  to  occur  only  in  a  restricted  portion  of  the  ultimate  flight  envelope.  This  step,  which 

3Hot  sections  contain  the  turbine  vanes,  blades,  and  combustor.  Cold  sections  contain  all  other 
parts,  including  the  shafts  and  disks  in  the  turbines. 

^The  Air  Force  maintains  a  CIP  for  e'’ch  engine  in  the  fleet  to  continue  development  activities 
after  the  engines  are  fielded.  The  CIP  allov.s  the  Air  Force  to  respond  to  problems  identified  during 
operation  and  to  improve  aspects  of  engines'  maintainability,  durability,  and  supportability.  The 
subsections  below  discuss  CIP  in  more  detail. 

®Air  Force  Regulation  800-30  currently  implements  this  four-step  process.  It  was  not  in  place 
during  the  developments  that  we  examine  here. 
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Table  3.1 

Baaic  Full-Scale-Development  (FSD)  Milestones  in  Airframe  and  Engine  Development 


Airframe 

19603/1970s  Engine  Doctrine 

Four-Step  Engine  Doctnne 

Preflight 

Preliminary  flight  rating  test 

Initial  flight  release  (IFR) 

demonstration 

(PFRT) 

Initial  flight 

Full  flight  release  (FFR) 

demonstration 

Full  fligh*' 

Military  qualification  test  (MQT) 

Initial  service  release  (ISR) 

demonstration 

Final  operational 

Operational  capability  release 

demonstration 

(OCR) 

SOURCES:  U.S.  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion.  1982,  chart  42; 
RadlofT,  1985,  charts  26-28. 


used  an  AMT  about  twice  as  long  as  the  flight  test  program,  just  to  be  safe,  corresponded  to 
the  old  PFRT  but  was  much  more  demanding. 

Full  flight  release  (FFR)  emphasized  performance  and  operability  and  opened  the  way 
for  a  full  exploration  of  the  flight  envelope.  It  also  used  an  AMT  about  one-eighth  to  one- 
quarter  the  full  expected  engine  life.  Using  such  an  AMT  at  this  point  reflected  less  the 
safety  requirements  to  explore  the  flight  envelope  than  an  expectation  that  engine  durability 
would  improve  in  tandem  with  other  aspects  of  the  engine  during  its  development. 

Durability  was  expected  to  reach  one-eighth  to  one-quarter  expected  engine  life  by  the  time 
FFR  was  approved. 

Initial  ser%ice  release  (ISR)  prepared  the  engine  for  low-rate  productioi  -md  used  an 
AMT  of  about  half  the  expected  engine  life.  It  corresponds  in  many  ways  to  the  old  MQT  and 
essentially  covered  all  the  ground  that  the  old  test  covered.  The  AMT  applied,  of  course,  was 
much  more  demanding.  Another  important  difference  is  the  relation  of  this  step  to  the 
production  decision.  Because  the  old  MQT  cleared  the  way  for  full-rate  production,  long-lead 
production  decisions  had  to  be  made  well  in  advance  of  this  milestone.  Under  the  four-step 
process,  long-lead  production  decisions  occurred  only  after  ISR,  significantly  reducing  the 
concurrency  between  development  and  production  present  in  the  older  system. 

Operational  capability  release  (OCR)  prepared  the  engine  for  full-rate  production  and 
used  an  AMT  for  full  expected  engine  life.  The  new  approach  essentially  added  this  step  to 
the  process  to  reflect  the  Air  Force’s  new  interest  in  durability.  To  get  from  ISR  to  OCR,  an 
engine  essentially  had  to  achieve  supportability  and  durability  goals  that  had  not  been 
important  in  the  past.  That  is,  the  Air  Force  had  traditionally  used  a  Component 
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Improvement  Program  (CIP)  to  identify  and  solve  such  problems  after  engines  were  fielded, 
The  CIP  had  played  and  continues  to  play,  an  important  role  in  impro-^ng  the  operation  and 
support  aspects  v-  f  all  Air  Force  engines.  By  introducing  the  OCR  step,  the  Air  Force 
attempted  to  solve  early  some  of  the  problems  normally  handled  in  the  CIP  and  thereby 
reduce  the  cost  of  retrofitting  solutions  to  those  problems  into  engines  in  the  field. 

The  Air  Force  has  assigned  a  carefully  structured  set  of  evaluation  tasks  to  each  step. 
Among  other  reasons,  they  are  assigned  so  that  problems  identified  late  in  the  process  tend 
not  to  require  solutions  that  would  require  the  Air  Force  to  repeat  tests  conducted  eaLrlier. 
This  structure  allows  the  Air  Force  to  eliminate  the  risks  inherent  in  any  engine  under  full- 
scale  development,  in  an  orderly  and  cost-eftective  manner. 

When  the  subject  developments  began,  this  structure  was  not  fully  >n  place.  The 
second  step  (FFR)  played  only  a  limited  role  in  the  derivative  developments  that  we  consider, 
in  any  case,  because  the  design  of  these  engines  was  well  enough  understood  so  that  little 
delay  was  required  between  the  first  and  second  steps  of  the  structure.  The  developments  we 
examine  expanded  the  flight  envelope  cautiously  and  methodically,  but  the  risks  associated 
with  that  expansion  were  not  considered  high  enough  at  the  time  to  warrant  a  full  s  -  cond 
step  in  the  development  process.  The  fourth  step  (OCR),  on  the  other  hand,  was  critical  to 
these  developments.  Because  that  step  gave  prominent  attention  to  the  need  for  greater 
durability,  we  should  expect  development  efforts  aimed  at  achieving  and  demonstrating 
durability  to  be  important. 

ENGINE  STRUCTURAL  INTEGRITY  PROGRAM 

Durability  is  an  issue  because  components  fail;  a  body  of  methods  that  allows  a 
designer  to  reduce  the  probability  and  cost  of  failure  holds  the  key  to  extending  an  engine’s 
life  and  reducing  its  operating  cosis.  The  engine  structural  integrity  program  (ENSI.F),  the 
third  technique  for  improving  technical  risk  assessment,  focuses  on  durability.  ENSIP  is  the 
distillation  of  a  set  of  design  ideas  that  Air  Force  developers  had  been  thinking  about  during 
the  1970s.®  The  Air  Force  was  in  the  process  of  formalizing  this  program  as  the  two 
developments  began.  As  c  result,  ENSIP  had  very  different  effects  on  different  parts  of  the 
development  that  wa  study  here. 

Simply  stated,  ENSIP  represents  a  basic  change  in  the  way  designers  think  about  the 
components  in  an  engine  system.  Before  ENSIP,  designers  implicitly  assumed  that 
cor,;ponents  could  be  designed  so  that  thej'  would  not  fail.  ENSIP  recognized  that  all 

CThis  discussion  draws  heavily  on  “Trends  in  Engine  Qualification"  (U.S.  Air  Force, 

Aeronautical  Systems  Division  Deputy  for  Propulsion,  1982). 
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components  can  fail  and  set  up  methods  to  explore  failure  modes,  consequences  of  failure, 
design  changes  that  could  reduce  the  probability  and  cost  of  failure,  and  maintenance 
procedures  that  could  prevent  catastrophic  failures.  Viewed  in  this  way,  the  ideas  embodied 
in  ENSIP  were  central  to  the  developments  we  study. 

As  engine  development  proceeded  in  the  early  1970s,  the  Air  Force  gave  relatively  less 
attention  to  aero-thermo  design  and  thrust  and  more  attention  to  engine  structure, 
materials,  and  management.  These  engine  developments  offered  a  fairly  consistunt  set  of 
lessons  learned.  Pepresentative  lessons  include  the  following;^ 

•  Identify  critical  parts — parts  whose  failure  jeopardizes  flight  safety — and  their 
potential  failure  modes  early 

•  Adequately  characterize  the  fracture  properties  of  materials  and  processes 

•  Do  not  assume  defect-free  structure  when  assessing  the  safety  of  flight 
components 

•  Verify  analytically  predicted  stresses  for  complex  components  with  formal 
empirical  tests 

•  Identify  internal  thermal  and  vibratory  environments  early  in  development 

•  Define  potential  engine/airframe  structural  interactions 

•  Reflect  the  anticipated  service  usage  of  the  engine  in  stress,  component,  and  full- 
scale  engine  tjsts 

•  Use  closed-loop  force  management  procedures  to  define  and  enforce 

—  realistic  inspection  and  maintenance  requirement.^ 

—  individual  engine  ti'acking  procedures 

—  deficiency  reporting 

—  updates  in  procedures  based  on  actual  usage. 

At  the  same  time,  a  seritj  of  studies  began  to  recommend  a  more  comprehensive  Air 
Force  approach  to  improving  its  development  of  engines.  For  example,  in  1976,  the  Air  Force 
Scientific  Advisory  Board  called  for  greater  emphasis  on  “engine  mechanical  and  structural 
integrity  and  durability”  and  “durability  and  damage  tolerance  assessments  ...  analogous  to 
those  being  performed  on  several  weapon  system  airframes."  Similarly,  in  1980,  the 
Comotroller  General  recommended  to  Congress  that  engine  development  be  complete  before 

■^U.S  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion,  1982,  charts  8~9. 
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production  arid  that  gas  turbine  developers  address  "reliability,  maintainability,  and 
durability  as  well  as  perfoimance."® 

ENSIP  was  a  formal  response  to  these  studies  and  to  lessons  learned  from  increasingly 
sophisticated  efforts  to  develop  engines  in  the  1970s.  It  effectively  implemented  the 
recommendations  of  the  1976  Scientific  Advisory  Board.  It  provided  “an  organized  and 
disciplined  approach  to  the  structural  design,  analysis,  development,  production,  and  life 
management  of  gas  turbine  engines  with  the  goal  of  ensuring: 

a)  engine  structural  safety 

b)  increase[d]  service  readiness,  and 

c)  reduce[d]  life  cycle  coats"® 

To  do  so,  ENSIP  employed  a  profile  of  use  over  the  engine’s  lifetime,  like  that  used  in 
the  AJVIT,  as  the  basis  for  ascertaining  an  individual  engine  component’s  performance  over  its 
lifetime.  How  long  components  last  defines  their  durability;  how  well  they  perform  under  the 
high  stresses  internal  to  the  engine  defines  their  damage  tolerance.  The  Air  Force  used  a 
technique  called  a  durability  and  damage  tolerance  assessment  (DADTA)  to  identify  typical 
structural  failure  modes  associated  with  each  engine  component;  the  effects  of  failure 
modes — in  particular,  whether  they  affected  flight  safety,  how  each  failure  mode  occurred 
over  time  and  how  it  could  be  predicted,  and  how  specific  failures  could  be  prevented  or 
managed  to  preserve  flight  saifety  and  engine  durability.  DAD” A  presupposed  engineering 
analysis  to  predict  where  and  when  failures  would  occur  in  an  engine  and  an  effective 
empirical  test  regime  in  which  to  test  these  predictions.  The  AMT  provided  that  test  regime. 

For  example,  r^NSIP  initially  attempted  to  design  the  cold  parts  of  an  engine  with  a 
lifo  equal  to  that  of  the  airframe  and  hot  parts  with  a  life  equal  to  half  of  that.  It  then  used 
DADTA  methods  to  determine  whether  these  goals  are  viable.  ENSIP  developed  a  set  of 
component  lives  that  could  be  sustained. 

This  process  gave  special  attention  to  critical  parts  that  might  fracture  during 
operation.  Fractures  generadly  develop  from  initial  flaws  whose  progress  can  potentially  be 
monitored.  The  analytic  process  determines  how  far  such  flaws  can  progress  without 
inducing  a  fracture  and  how  long  such  progress  takes,  in  cycles  or  hours;  this  time  becomes 
the  design  life  for  a  critical  part.  The  process  then  schedules  an  inspection  to  occur  half  way 

8U.S,  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion,  1982,  charts  4—5. 

9U.S.  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion,  1982,  chart  36. 
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to  this  design  life  for  each  critical  part  in  the  engine.  This  process  yields  damage-tolerant 
design. 

The  ENSIP  approach  and  its  constituent  techniques  could  first  be  applied  in  the 
design  of  a  new  engine  and  then  used  repeatedly  through  the  engine’s  lifetime.  Table  3.2 
presents  the  Propulsion  SPO’s  1982  view  of  the  range  of  tasks  that  might  be  included  in  this 
loop. 

At  the  beginning,  the  approach  provided  a  way  to  set  specifications  on  engine 
durability,  reliability,  availability,  and  maintainability.  It  provided  un  analytic  approach  to 
designing  against  such  specifications  and  testing  as  the  designs  proposed. 


Table  3.2 

The  Five  ENSIP  Tasks 


I.  Design  Information 

EN.SIP  maste'"  plan 

desif,Ti  service  life  and  usage  requirements 
design  criter.u 

II.  Design  Analysis  of  Components  and  Material  Characteristics 

design  duty  cycle 

materials  and  processes  design  data  charactenzed 
structural  thermal  analysis 
manufacturing  and  quality  control 

III.  Component  and  Core  Engine  Testing 

strength  testing 
damage  tolerance  tests 
durability  tests 
thermal  survey 

\ibratory  strain  and  flutter  boundary  .••  r\’ey 

IV.  Ground  and  Pligh*'  Engine  Tests 

environment  verification  testing 
AMT  test  specification  derived 
durabilit:,'  tests  (AMT) 
damage  ’^^olerance  tests 
flight  test  strain  survey 

updated  durability  and  damage  tolerance  control  plan 
perform  deterioration  structural  .'mpact  assessment 
critical  part  update 

V.  Production  Quality  Control  and  Engine  Life  Management 

production  engineering  analysis 
structural  safety  and  durability  summary 
engine  structure  maintenance  plan 
individual  engine  tracking 
Lead  the  Force  program  (usage) 

durabili'.y  and  damage  tolerance  control  plan  implemented 
technical  order  update 

SOURCE:  U  S.  Air  Force,  Aeronautical  Systems  Division  Deputy  for 
Propulsion,  1982,  cha't40. 
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But  the  process  need  not  and  should  not  stop  when  the  engine  was  fielded.  Linked  to  a 
good  information  system  on  the  experience  of  individual  engines,  this  approach  provided  a 
framework  for  managing  problems  identified  during  field  operations  and  bringing  them  into 
the  CIP  for  resolution  or  amelioration.  If  mission  profiles  changed,  it  allowed  a  way  to 
predict  the  effects  on  ari  engine  and  the  maintenance  and  redesign  efforts  that  should  be 
undertaken  in  anticipation  of  such  a  change.  If  new  materials  or  component  designs  became 
available,  it  provided  a  way  to  predict  and  rest  the  likely  effects  of  exploiting  such 
innovations  and  adjusting  maintenance  policy  in  anticipation  of  their  introduction. 

That  is,  such  activities  closed  the  engine-management  loop  by  continually  bringing 
operational  information  back  to  the  ongoing  development  process  and  using  the  best 
operational  information  available  to  inform  ongoing  development  under  the  CIP  for  the 
engine  in  question  or  other  efforts  to  develop  follow-on  engines. 

ENSIP  was  working  its  way  from  a  proposal  to  a  formal  .set  of  Air  Force  regulations 
when  our  developments  began.  As  formal  regulations  implementing  ENSIP  neared 
completion,  the  Propulsion  SPO  looked  for  ways  to  implement  the  basic  ideas  underlying 
ENSIP  in  as  many  of  its  engine  development  activities  as  possible.  It  expected  to  implement 
all  aspects  of  ENSIP  in  such  new  designs  as  the  F109  and  advanced  tactical  fighter  (ATF). 

For  established  engines  like  the  FlOO  and  TF34,  the  Propulsion  SPO  expected  damage 
tolerance  assessments  and  AMTs  to  yield  better  estimates  of  component  lives,  improved 
maintenance  and  repair  plans,  and  better  improvements  of  new  parts  developed  in  the  CIP. 
The  Propulsion  SPO  planned  to  implement  ENSIP  on  a  more  limited  basis  in  the  FlOl,  FI  10, 
and  FlOO  improvement  developments.  For  these,  it  plarmed  to  use  DADTA  and  Ajf4T 
methods  to  improve  their  maintenance,  repair,  and  support  plans. 

SUMMARY 

The  management  of  system  development  is  risk  management.  System  development  is 
inherently  risky;  experienced  meuiagers  seek  a  low-risk  system  design  and  then  try’  to 
minimize  the  risks  associated  with  development  based  on  this  design.  While  doing  so,  they 
must  continually  balance  risks  associated  with  the  technology  development  itself  against 
external  risks  associated  with  political  and  funding  support  for  the  program.  Innovations  in 
the  forma]  methods  available  to  the  Air  Force  in  the  late  1970s  improved  its  ability  to 
manage  the  FlOO-PW-220  and  FllO-GE-100  developments  and  subsequent  programs  as  well. 

These  innovations  include  the  AMT,  a  rigorous,  empirictuiy  based  durability  test;  the 
four-step  process,  which  reduced  concurrency  between  engine  development  and  production 
and  increased  the  importance  of  operability,  supportability,  and  durability  in  engine 


-27- 


development;  and  ENSIP,  a  set  of  formal  methods  for  understanding  component  failures  and 
changfing  design  and  maintenance  policies  to  reduce  the  probability  and  cost  of  stich  failures. 

All  these  innovations  played  a  role  in  ensuring  the  success  of  the  developments  we 
examine  here.  In  the  end,  however,  risk  management  cannot  be  separated  from  the 
experienced  manager  and  his  or  her  judgment.  The  creation  of  the  Propulsion  SPO  promoted 
continuity  from  one  engine  development  to  the  next,  bringing  important  lessons  learned  from 
the  past  to  bear  on  the  FlOO-PW-220  and  FllO-GE-100  developments,  and  contributing 
significantly  to  the  Propulsion  SPO’s  reputation  as  an  organization  that  assesses  and 
manages  risk  well. 
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4.  A  BRIEF  HISTORY  OF  THE  F110-GE-100  AND  F100-PW-220  DEVELOPMENTS 


The  FlOO-PW-220  and  FllO-GE-100  are  best  known  as  the  en^nes  that  brought 
competition  to  the  Air  Force  in  the  Great  Engine  War  of  the  early  1980s.  ‘  More  formally, 
this  competition  was  called  the  Alternate  Fighter  Engine  competition  between  Pratt  and 
Whitney  (P&W)  and  General  Electric  (GE). 

In  the  mid-1970s,  P&W  dominated  the  military  engine  business.  It  had  triumphed 
over  its  primary  competitor,  GE,  in  the  source  selection  to  provide  engines  for  the  F-15 
fighter  in  1970,  and  the  Air  Force  selected  a  minor  variation  on  the  same  engine  for  its  F-16 
fighter,  introduced  in  1978.  These  were  the  FlOO-PW-lOO  and  FlOO-PW-200  engines, 
respectively,  and  became  the  basis  for  P&Ws  FIDO  “family"  of  engines.  GE  won  the  source 
selection  to  provide  engines  for  the  B-1  bomber,  but  that  procurement  was  subsequently 
canceled  in  1977  before  production  began,  leaving  GE  with  little  military  engine  business. 

The  first  “skirmishes"  of  the  Great  Engine  War  effectively  began  when  GE  endeavored 
to  use  the  engine  it  had  developed  for  the  B-1,  the  FlOl-GE-lOO,  as  the  basis  for  a  fighter 
engine.  GE  initially  targeted  the  market  for  reengining  the  Navy  F-14  but,  on  the  basis  of 
Air  Force  interest,  shifted  its  emphasis  to  include  the  F-16  and  then  the  F-15.  That  effort  led 
to  the  development  of  the  FllO-GE-lCO.  In  the  meantime,  P&W  tried  to  stop  that 
development  effc  jfore  it  could  yield  a  feasible  alternative  to  its  FlOO  engines,  but 
ultimately  countered  with  an  engine  based  on  its  own  FlOO,  the  FlOO-PW-220. 

This  section  outlines  the  origination  process  for  the  FllO-GE-100  and  FlOO-PW-220 
engines.  It  presents  the  key  facts  in  the  Air  Force’s  efforts  to  develop  two  derivative  fighter 
engines  that  could  then  contend  against  one  another  in  its  Alternate  Fighter  Engine 
production  competition  in  1984  and  in  subsequent  years.  It  describes  the  basic  engines 
involved,  the  development  tasks  required  to  reach  the  final  competing  engines,  and  the 
contracts  used  to  implement  those  developments.  It  also  describes  the  basic  Air  Force 
environment  in  which  the  developments  occurred  and  the  resources  required  to  complete 
them. 

THE  INITIAL  SETTING 

When  P&W  designed  the  FlOO  for  the  Air  Force,  the  Air  Force  emphasized  a  dramatic 
increase  in  thrust,  a  change  from  the  4:1  thrust-to-weight  ratio  in  the  best  fighter  engines 

•For  useful  general  discussions  cf  this  competition,  see  Drewes,  1987;  Kennedy,  1985;  Mayes, 
1968;  and  Ogg,  1987.  This  section  draws  on  all  these  sources,  especially  Drewes. 
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preceding  the  FIDO  to  8:1.  Early  in  the  development,  P&W  engineers  noted  that  the  engine 
they  were  developing  would  be  expensive  to  maintain.  Air  Force  officials  told  them  not  to 
worry  about  that  then;  that  was  something  they  could  address  once  the  anticipated  increase 
in  thrust  was  achieved.^  It  was.  The  advance  was  so  great  that  Air  Force  fighters  for  the 
first  time  had  a  thrust-to-weight  ratio  of  greater  than  one,  allowing  pilots  to  change 
dramatically  the  tactics  they  had  used  in  the  past.  One  result  was  that  pilots  used  many 
more  engine  cycles— essentially,  throttle  movements  from  low  power  to  high  power  and 
back — per  flight  hour  than  they  had  in  the  past.  This  change  placed  even  more  stress  on  the 
engines  than  the  designers  had  anticipated,  increasing  maintenance  costs  and  reducing  the 
engines'  effective  lives.  In  sum,  the  Air  Force  had  a  powerful  new  engine  that  was  extremely 
expensive  to  use. 

The  FiOO  had  an  additional  problem.  In  certain  flight  conditions,  it  experienced  stall- 
stagnations:  “An  engine  stall  is  a  momentary  hesitation  in  engine  operation  caused  by  a 
disturbance  in  the  air  flow  and  resulting  in  an  aerodynamic  stall  of  the  compressor  blades.  If 
the  stall  is  not  self-recovering  through  immediate  adjustment  of  the  air  flow,  it  is  called  a 
stagnation.  The  pilot  must  shut  down  the  engine  and  restart  it.”^  F-15  pilots  could  readily 
recover  with  two  engines  and  at  high  altitudes.  But  this  problem  made  flight  at  lower 
altitudes  or  under  pressure  in  combat  unacceptably  unsafe.  And  the  Air  Force  expected  the 
problem  to  increase  for  F-16  pilots,  who  would  have  only  one  engine  to  rely  on. 

The  Air  Force  turned  to  P&W  to  fix  both  these  problems.  P&W  pointed  out  that  it  had 
met  the  Air  Force's  specifications  for  the  engine  and  had  in  fact  warned  the  Air  Force  about 
the  likely  problem  of  high  operating  cost.  P&W  stood  ready  to  help  the  Air  Force  resolve 
these  problems,  but  only  if  the  Air  Force  was  prepared  to  compensate  it  fully  for  its  work. 

Pratt  and  Whitney’s  firm  stand  and  aggressive  negotiating  style  during  a  period  when 
it  was  manufacturing  an  engine  that  many  in  the  Air  Force  felt  was  inadequate  increasingly 
alienated  officials  in  the  Air  Force.  Many  of  those  officials  came  to  regret  P&Ws  dominant 
position  in  the  fighter  engine  business,  believing  that  P&W  would  not  be  so  arrogant  and 
unresponsive  if  it  faced  a  viable  competitor.  Dissatisfaction  with  P&W  deepened  and  spread 
through  the  Air  Force  as  time  passed  with  no  resolution  of  the  problems.  Additional  support 
for  finding  a  competitor  grew  when  a  number  of  key  P&W  subcontractors  went  on  strike  in 
1979,  endangering  the  Air  Force’s  ability  to  field  new  F-15s  and  F-16s  as  their  manufacturers 
completed  their  airframes. 


2For  details,  see  Ogg,  1987. 
^Drewes,  pp.  51-62. 


•  30  • 


Meanwhile,  GE  was  looking  for  an  implementation  of  the  technology  it  had  developed 
for  the  F101>GE-100  engine.  GE  had  a  long-standing  business  philosophy  of  building 
families  of  engines  based  on  similar  cores.  If  GE  could  use  the  core  components  it  de  jloped 
in  more  than  one  engine,  it  could  effectively  recover  its  development  costs  more  easily  and 
use  learning  in  production  to  drive  down  the  costa  of  producing  those  components  more 
rapidly.  These  factors  would  make  it  more  competitive,  especially  in  the  civilian  aviation 
industry  where  it  had  a  significant  foothold.  GE  was  marketing  the  CFM-56  engine,  ait 
engine  closely  related  to  the  FlOl,  to  the  private  sector.  If  it  could  find  a  mili'  nry  market  for 
that  engine  or  use  of  its  core  components  in  a  related  engine,  it  could  increase  its 
competitiveness. 

GE’s  first  target  for  a  military  engine  was  a  fighter  engine  to  replace  the  TF30  engines 
in  the  existing,  under-powered  Nnvy  F-14.  In  1975,  GE  assembled  a  demonstrator  from 
components  used  in  or  based  on  its  FlOl  and  F404  engines.  GE  sized  this  demonstrator, 
which  became  known  as  the  FIOIX,  to  power  the  F-14,  But  the  Navy  did  not  place  a  high 
enough  priority  on  reengining  the  F-14  to  pay  for  its  likely  billion-dollar  cost.  GE  turned  to 
the  Air  Force.  The  Air  Force  was  impressed  enough  to  continue  monitoring  GE’s  work.  The 
Air  Force  also  initiated  several  small  studies  in  its  own  laboratories  to  explore  several 
technical  issues  associated  with  developing  the  FlOlX  further. 

The  Air  Force  was  particularly  impressed  by  the  fact  that  the  FlOlX  seemed  to 
address  the  two  big  problems  it  had  with  the  FlOO:  operating  costs  and  stall-stagnations. 
First  the  FlOlX  incorporated  a  long-life  core  section  from  the  FlOi  engine  that  should  have 
dramatically  reduced  operating  costs.  When  the  B-1  mission  changed  to  require  low-altitude 
flight,  the  number  of  cycles  expected  on  the  FlOl  engines  increased  dramatically  and,  hence, 
durability  became  a  high  priority  for  the  engines.  When  the  B-1  was  canceled,  GE  engineers 
used  what  they  hoped  was  a  hiatus  to  continue  increasing  the  durability  of  the  engine  core. 
Such  a  core  could  help  address  one  of  the  key  problems  that  the  Air  Force  was  then 
experiencing  with  the  FlOO  engine.  Second,  GE  used  a  closed-loop  control  system  that 
effectively  eliminated  stall-stagnations.  As  Air  Force  aggravation  with  P&W  grew,  the 
FlOlX  looked  increasingly  attractive  as  a  viable  tlireat  that  the  Air  Force  could  use  to  medte 
P&W  more  responsive. 

By  1979,  the  Air  Force  was  ready  to  fund  GE  to  present  just  such  a  threat.  An  Air 
Force-Navy  memorandum  of  understanding  provided  initial  funding  for  engine  development 
in  a  way  that  heavily  favored  GE.  The  Air  Force  justified  this  imbalance  at  the  time  in 
testimony  to  Congress,  which  stated  that 
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Pratt  will  be  working  from  a  development  funding  boee  on  the  FlOO  and  TF30  enginee  in 
excel!  of  $2  biiliun  and  on  experience  bate  in  afterburning  turbofan  fighter  enginee  of  14 
years.  GE  will  be  working  from  a  development  funding  base  on  the  FlOl  of  $0.7  billion 
with  no  previous  experience.* 

That  statement  quite  succinctly  summarizes  the  circumstances  at  the  beginning  of  the  two 
developments  we  examine  here. 

THE  GE  F101  DFE  AND  FHO-GE-IOO 

Air  Force  support  for  competition  coalesced  in  stages.  Table  4.1  summarizes  the  msuor 
events  in  the  competitive  developments.  Development  began  in  an  Engine  Model  Derivative 
Program  (EMDP)  in  1979  that  completed  the  initial  design  for  the  engine  based  on  the  GE 
FIOIX  and  began  initial  testing.  After  the  EMDP  began,  the  Air  Force  expanded  it  to 
perform  more  extensive  development  and  design  work.  When  the  work  verified  that  GE 
could  in  fact  provide  an  alternative  to  the  FlOO.  the  Air  Force  added  funding  to  keep  the 
design  team  in  place  while  it  considered  whether  to  initiate  a  formal  competition.  By  1982, 
the  .\ir  Force  was  committed  to  a  competition  and  issued  a  full-scale  development  contract 
for  the  new  GE  ent.ine,  which  would  be  known  as  the  FllO-GE-100.  In  1985,  the  Air  Force 
extended  this  effort  to  include  a  number  of  post-qualification  tasks.  As  production  engines 
became  available,  the  Air  Force  set  up  an  accelerated  operational  testing,  or  "Lead  the 
Force,"  program  to  check  performance  in  the  field.  It  also  set  up  the  Component 
Improvement  Program,  which  DoD  customarily  employs  to  handle  problems  in  engines  that 
come  to  light  after  they  are  introduced.  This  section  reviews  each  stage  of  the  development 
program  for  the  FI  10  engine. 

Tha  Engine  Model  Derivative  Program 

The  first  clear  step  in  development  occurred  in  1979,  when  the  Air  Force  was  looking 
for  an  opportunity  to  initiate  an  Engine  Model  Derivative  Program.^  The  FlOlX  looked  like 
a  good  candidate.  An  EMDP  could  be  executed  for  a  small  cost  to  the  Air  Force  and  at  fairly 

^Quoted  in  Drewes,  1987,  pp.  101-102. 

®EMDP  was  based  on  the  Component  Improvement  Program,  a  class  of  development  effort  that 
the  Air  Force  had  used  to  respond  to  operational  problems  in  fielded  engines.  Over  time,  the  Air  Force 
used  its  annual  CEP  program  more  and  more,  not  just  to  realize  the  expected  performance  of  existing 
engines  after  they  were  fielded,  but  to  improve  the  performance  expected  from  them.  Congress  saw 
such  efforts  as  an  attempt  to  end-run  its  oversight  authority  on  new  engines.  In  1968,  Congress  created 
the  EMDP  vehicle  as  a  formal  device  that  allowed  such  derivative  development,  but  gave  it  more  direct 
visibility.  The  Air  Force  did  not  actually  use  the  vehicle  until  1979. 
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TabU  4.1 

Timeliiie  for  F110-GE*100  and  F100>FW-220  Developmonta 


Date 

FVoduction 

Competition 

FllO-GE-100 

Development 

FlOO-PW-220 

Development 

1979 

Feb 

Jul 

FlOl  DFE  ElMDP  begins 

FlOO  EMDP  begins 

1980 

Apr 

Testing  begins 

1981 

Jan 

Jun 

Aug 

Air  Staff  hae  acquiaition 
strategy  for  competition 

RFI  issued 

EMDP  expands 

1982 

May 

Sep 

Nov 

Dec 

Draft  RFP  issued 

TED  begins 

FSD  begins 

FSD  begins 

1983 

May 

Jun 

Dec 

Final  RFP  issued 

Best-and'fmal  offers 

-220  specification  called  for 

1984 

Feb 

Source  selection 

1986 

Jan 

Feb 

May 

Jul 

FYoduct  Validated 

First  production  engine 

CED  begins 

DEEC/GP  qualified 

GP/BPV  qualified 

ILC  upgrade  qualified 

Nov 

-220  product  validated;  first 
production  engine 

1986 

Aug 

(n.d.l 

First  delivery  to 
operation^  unit 

First  delivery  to  operational 
unit 

low  risk,  demonstrating  the  value  of  the  EMDP  concept.  And  it  offered  the  potential  benefit 
of  threatening  P&W  with  competition  and  thereby  simplifying  the  Air  Force’s  problem  of 
managing  its  most  important  engine  at  the  time,  the  FIOO.  Despite  intense  opposition  from 
P&W,  the  Air  Force  acquired  the  funds  needed  to  initiate  this  demonstration.  By  February 
1979,  a  30-nionth  firm-fixed-price  contract  for  $79.7  million  was  in  place  for  the  FlOl  DFE 
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(Derivative  Fighter  Engine)  EMDP.  The  Air  Force  was  the  lead  service  in  this  joint  program 
with  the  Navy  and  dominated  the  development  throughout.^ 

The  contract  called  for  GE  to  “complete  the  initial  design  [for],  develop,  and  test  a 
Derivative  Fighter  Engine’  based  on  the  FlOl  engine  and  “using  the  FlOl  core  engine  design 
to  the  full  extent  possible....  A  primary  goal  [was]  to  define  and  develop  the  Derivative 
Fighter  Engine  to  the  point  where  a  flight  test  demonstration  [could]  be  safely  and 
economically  conducted  in  F-16  and  F-14  aircraft.  An  equally  important  goal  [was]  to  provide 
a  base  of  design  knowledge  and  test  experience  with  the  engine  design  to  form  a  basis  for 
decision  making  on  future  engine  development  and  production  options  for  the  USAF  and 
USN.”  This  “limited  development  program’  called  for  at  least  1,000  engine  ground-test  hours 
and  the  refurbishment,  acceptance  testing,  and  delivery  of  three  nameplate  engines  to  the 
Air  Force  for  flight  testing.  “At  the  conclusion  of  the  Limited  Development  flight  test 
program,  ...  [GE  was  to]  submit  engine  characteristics  [that  GE  would  guarantee  during  full 
scale  development,  including]  engine  life,  maintenance  parameters,  performance,  reliability, 
and  cosL”^ 

GE  came  to  this  program  with  a  well-articulated  design.  The  proven  FlOl  core 
rt'quired  little  further  work.  In  early  1979,  an  Air  Force  review  of  risk  factors  associated 
with  controls  and  accessories,  augmentor,  high-cycle  fatigue,  and  inlet  compatibility 
concluded  that  the  use  cf  many  well-characterized  engine  components,  a  well-characterized 
inlet,  and  a  logical  pre-flight  testing  sequence  ensured  a  low-risk  F-16  flight  test  evaluation. 
And  Air  Force  planners  expected  the  FlOTs  ruggedness  to  help  them  control  some  of  the 
higher  risks  normally  present  in  an  engine  test  program.  The  primary  tasks  remaining  in 
the  engine  design  were  characterization  of  the  design  in  the  full,  anticipated  flight  envelopes 
of  the  F-14  and  F-16  and  refinement  of  software,  parameter  settings,  and  other  aspects  of  the 
design  that  remained  variable  in  a  fixed  hardware  design. 

At  first  glance,  it  might  appear  that  developing  a  new  engine  for  the  one-engine  F-16 
would  be  riskier  than  developing  one  for  the  two-engine  F-15,  which  could  always  fly  with 
one  proven  engine  while  testing  the  new  engine.  Unfortunately,  GE’s  FlOl  DFE  would  not 

®The  principal  difference*  in  the  goals  of  the  Air  Force  and  those  of  the  Navy  at  the  beginning  of 
this  development  were  in  the  airflows  they  sought  and  resulting  differences  in  the  control  schedules 
they  needed.  The  Navy  wanted  to  exploit  the  full  potential  of  the  engine;  the  Air  Force  clicse  initicdly 
not  to.  To  deal  with  these  differences,  each  service  depended  on  its  contractors.  Most  interservice 
coordination  took  place  through  GE,  General  Dynamics  (the  F-16  prime  contractor),  and  Grumman  (the 
F-14  contractor). 

■^.S.  Air  Force,  Aeronautical  Systems  Division,  Letter  Contract  F33657-79-C-0176,  Wright- 
Patterson  Air  Force  Base,  Ohio,  26  January  1979,  Attachment  1,  pp.  1-8.  This  language  is  consistent 
with  that  in  the  defmitized  contract,  issued  as  Modification  PZ0003  of  Contract  79-C-0176  on  16  July 
1979. 
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fit  in  the  F-15.  Developers  expected  risks  to  be  higher  if  they  attempted  to  alter  the  engine  to 
fit  the  F-15  than  if  they  used  a  simpler  test  engine,  closer  to  earlier  proven  designs,  in  the 
one-engine  F-16.  For  a  similar  reason,  early  efforts  with  the  FlOl  DFE  did  not  contemplate 
any  changes  in  the  F-16  inlet.  Changes  from  earlier  designs  were  kept  to  a  minimum  to 
control  risk  in  the  early  stages  of  development.  Only  later,  as  the  design  matured  and  proved 
itself,  did  developers  adapt  the  FlOl  to  the  F-15  and  optimize  aircraft  inlets  for  the  new 
engine. 

During  the  course  of  the  EMDP,  the  Air  Force  expanded  its  work  scope  to  include 
construction  of  an  additional  engine  that  GE  could  use  to  ‘'improve  the  operability,  life,  and 
performance  balance  of  the  FlOl  Derivative  Fighter  Engine  limited  development 
design  .  .  .  based  on  the  experience  gained  during  the  limited  development  program.  Engine 
design  faults  identified  in  both  the  ground-  and  flight-test  programs  [were  to]  be  addressed 
and  eliminated  where  possible.  No  high-risk  design  changes  [were  to]  be  made  to  the  limited 
development  design."  As  part  of  this  expanded  work  scope,  the  Air  Force  required  explicit 
but  limited  structural  durability  and  damage  tolerance  assessments  and  additional 
durability  testing.®  The  Air  Force  also  contracted  for  design  work  to  make  the  FlOl  DFE 
compatible  with  the  F-15  ®  This  effort  initiated  a  significant  redesign  effort  in  the  context  of 
this  development,  which  yielded  a  “slimline  engine"  with  a  profile  compatible  with  the  F-15 
engine  bay,  which  required  a  smaller  engine  than  the  F-14  and  F-16.  The  slimline  engine 
simply  changed  the  locations  of  some  auxiliary  engine  equipment.  These  changes  increased 
the  size  of  the  contract  to  $101.8  million. 

The  largest  problem  that  GE  and  the  Air  Force  faced  in  this  development  was  funding. 
In  part  because  P&W  attempted  repeatedly  in  the  Congress  and  elsewhere  to  derail  the 
development  effort,  the  FlOl  DFE  EMDP  operated  in  a  continuously  precarious  funding 
environment.  In  the  end,  however,  P&W s  tactics  produced  precisely  the  opposite  effect  from 
that  intended:  P&Ws  aggressive  efforts  to  cut  funding  to  GE,  together  with  its 
unwillingness  to  correct  problems  with  the  FlOO,  consolidated  support  for  the  FlOl  DFE 
within  the  Air  Force  and  probably  ensured  that  sufficient  funding  and  management  support 
would  be  available  to  complete  the  EMDP.*° 

Meanwhile,  other  events  increased  the  likelihood  that  GE's  efforts  would  yield  a  viable 
engine.  President  Reagan’s  totally  unrelated  decision  to  reinstate  the  B-1  program,  made  in 

®U.S.  Air  Force,  Aeronautical  Systems  Division,  Modification  P00017  to  Contract  79-C-0176, 
Wnght-Patterson  Air  Force  Base,  Ohio,  1  October  1980,  Attachment  1,  pp.  1-5. 

®U.S.  Ar  Force,  Aeronautical  Systems  Division,  Modification  P00020  to  Contract  79-C-0176, 
Wnght-Patterson  Ar  Force  Base,  Ohio,  14  August  1981,  Attachment  1,  p.  1. 

lODetails  of  P&W’s  efforts  to  stop  the  GE  effort  ore  wtl!  documented  in  Drewes,  1987. 
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February  1981,  and  the  Air  Force's  decision  around  the  same  time  to  use  the  CFM-66  in  the 
KC-136,  meant  that  production  runs  for  major  subsysto’^s  in  the  FlOl  DFE  would  be  much 
longer  than  expected,  substantially  reducing  the  cost  of  producing  the  FlOl  DFE.  In 
addition,  commonality  in  tooling  for  these  engines  would  cut  the  cost  of  tooling  for  the  FllO 
by  more  than  50  percent.  Together,  these  considerations  reduced  the  production  costs  for  the 
i'llO  by  25  percent,  helping  it  to  become  a  viable  competitor  with  the  upgraded  FIOO." 

When  GE  successfully  completed  this  program  of  development,  it  was  clear  that  the 
FlOl  DFE  could  in  fact  provide  an  alternative  to  the  troublesome  FlOO  if  the  Air  Force 
wanted  one.  As  early  as  January  1981,  the  Air  Staff  established  a  fighter-engine  acquisition 
strategy  to  compete  the  FlOl  DFE  against  an  improved  FIOO.'^  An  active  debate  escalated 
within  the  Air  Force. How  much  would  it  cost?  What  engine  capabilities  would  the  two 
contractors  deliver  in  a  competition?  A  request  for  information  from  GE  and  P&W  issued  in 
August  1981  produced  some  useful  answers  to  these  questions,  but  Air  Force  decision  makers 
needed  more  time  before  they  would  commit  to  a  competition. 

Transition  to  Engineering  Development 

To  keep  the  FlOl  DFE  design  team  intact  while  deliberations  continued,  the  Air  Force 
extended  the  FlOl  DFE  EMDP  through  March  1983,  adding  $31.2  million  to  the  contract  to 
allow  additional  testing  and  to  prepare  for  a  potential  full-scale  development.  This 
preparation  included  the  design  of  a  common  configuration  for  engines  to  be  used  in  the  F-14, 
F-15,  and  F-16  that  could  be  qualified  for  production  in  full  scale  development.  To  achieve 
that  design,  GE  initiated  associate  contractor  agreements  with  McDonnell  Douglas  and 
General  Dynamics,  the  manufacturers  of  the  F-15  and  F-16,  and  initiated  the  development  of 
interface  control  documents  to  specify  the  details  required  to  integrate  this  engine  with  the 
two  aircraft.  This  effort  also  included  design  tasks  to  make  the  engine  producible,  including 
“cost  reductions,  weight  reductions,  and  corrective  actions  resulting  from  evaluation  of 
engine  flight  test  data.”  And  it  included  developing  a  “rationale  for  the  level  of  the  engine 
monitoring  system  to  be  developed  for  the  FlOl  DFE  engine."  The  Air  Force  called  this 
extension  the  Transition  to  Engineering  Development.*^ 

'*Drewes,  1987,  p.  145;  Kennedy,  1985,  p.  11. 

'20ur  description  relies  in  part  on  ofTicial  Air  Force  histories  of  various  Aeronautical  Systems 
Division  programs  written  by  C.  J.  Geiger  in  1981-1982.  Because  those  unclassified  histories  are 
chapters  in  classified  Air  Force  documents,  we  cannot  provide  explicit  citations  in  this  document. 

■^For  details,  see  Drewes,  1987;  Kennedy,  1985. 

*^U.S.  Ar  Force,  Aeronautical  Systems  Division,  Modification  P00022  to  Contract  79-0-0176, 
Wright-Patterson  Ar  Force  Base,  Ohio,  1  May  1982,  Attachment  1,  pp.  3—4. 


-36- 


Full'Scil*  D«valopm«nt 

In  October  1981,  the  Air  Forco  Council  on  Fighter  Engine  Acquisition  Strategy 
reccmmendod  qualification  programs  for  the  FlOl  PFE  and  FlOO  DEEC,  an  improved 
version  of  the  F100.‘^  By  the  end  of  1981,  the  Air  Force  had  concluded  that  a  competition 
made  sense.  The  Air  Force  worked  through  1982  to  build  support  for  the  idea  and  ultimately 
issued  a  draft  request  for  proposals  to  GE  and  P&W  in  December  1982.  That  action 
effectivaly  began  the  formal  Alternate  Fighter  Engine  production  competition.  In 
anticipation  cf  that  action,  in  November  1982,  the  Air  Foice  initiated  a  full-scale 
development  contract  for  the  production  version  of  the  FlOl  DFE,  the  FllO-GE-100.**  Under 
this  firm-fixed-price  contract,  GE  would  modify  cn  j  engine  from  the  EMDP  and  fabricate  two 
additional  engines  to  the  full-scale-development  specification  to  use  for  durability  and  other 
testing.  GE  would  then  "perform  those  development  activities  necessary  to  verify  that  the 
Flit'  engine  m^iets  the  requirements  of  the  FI  10  Prime  Item  Development  (PID) 
Specification.  .  .  .  [GE  would  also]  accomplish  the  necessary  studies  with  the  F-16  and  F-16 
contractors  to  assure  engine/airframe  compatibility."  The  contract  required  GE  to  assess  its 
new  engine  relative  to  the  standards  set  by  ENSIP  standards  and  to  develop  a  maintenance 
plan  based  on  tliose  standards.  It  also  required  GE  to  develop  an  engine  monitoring  system, 
using  existing  systems  to  the  fall  extent  possible.  GE  and  the  Air  Force  negotiated  a  price  of 
$109.3  million  for  this  work.*’ 

Development  ran  fairly  smoothly  on  this  contract.  GE  and  the  Air  Force  completed  the 
critical  design  review  on  the  production-verification  design  in  January  1983.  Construction  of 
the  first  slimline  engine  was  completed  shortly  thereafter.  Full-scale  development  consisted 
in  large  part  of  efforts  to  test  this  slimline  version,  and  to  constmet  and  qualify  final 
production-verification  engines.  The  Air  Force  and  GE  negotiated  a  series  of  small 
adjustments  in  the  work  scope  during  1983  and  1984,  but  for  the  most  part  development 
P'oceeded  as  planned.  The  primary  difficulty  was  a  persistent  tendency  for  GE  to  be  late  on 
the  test  plan  and  other  parts  of  the  development  plan  and  on  reports  and  other  deliverables; 
Section  5  discusses  this  problem  in  more  detail. 

'Welson,  1983a.  We  discuss  improved  FlOO  engines  below. 

'®The  EMDP  effort  did  not  stop  at  this  point.  It  continued  in  an  effort  to  demonstrate  and 
validate  concepts  that  would  lead  to  the  next  derivative  engine  based  on  the  FllO,  the  FllO-GE-129 
improved  performanci  '.ngine  (IPE).  That  engine  would  allow  engine  competition  to  continue  when  the 
Air  Force  procurement  of  the  FllO-GE-100  was  complete.  It  reflect-'d  the  Air  Force’s  continuing 
commitment  to  competition  for  engines. 

'’U.S.  Air  Force,  Aeronautical  Systems  Division,  Contract  F33657-82-C-0257,  Wright-Patterson 
Ail  Force  Base,  Ohio,  16  November  1982,  pp.  2,  4—6;  Attachment  1,  pp.  1,  8-9,  14. 
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In  tiie  meantime,  the  Ait  Force  issued  the  final  request  for  proposals  to  GE  and  P&W 
for  a  production  competition  in  May  1983  for  2,004  engines  for  F-lSs  and  F-16s  during  fiscal 
years  (FYs)  85-90.‘®  Among  its  many  stipulations,  the  request  required  a  warranty  with 
clearly  defined  and  stringent  terms  that  essentially  shifted  the  risk  of  failing  to  meet  stated 
performance  goals  to  the  contractor.  GE  si  bmitted  a  best-and-final  offer  in  December  1983. 
And  in  December,  acting  on  a  request  from  the  Office  of  the  Secretary  of  Defense  earlier  in 
the  year,  the  Defense  System  Acquisition  Review  Committee  (DSARC)  reviewed  the  proposed 
production  competition.  In  its  approval,  it  noted  the  importance  of  the  planned  warranties  to 
the  management  of  remaining  risk  in  the  development  efforts.  In  February  1984,  the  Air 
Force  awarded  production  contracts  for  FY  85,  giving  GE  an  award  for  120  engines  in  F-16s, 
or  75  percent  of  the  total  awarded  in  the  competition.  That  award  proi'ided  the  first  tangible 
evidence  that  a  market  would  exist  for  the  engine  that  GE  was  verifying  in  full-scale 
development.  Three  days  later,  the  Navy  chose  a  close  variation,  the  FllO-GE-4^0,  to 
reengine  the  F-14.  A  number  of  foreign  governments  quickly  followed  suit. 

Continuing  Engineering  Development 

The  plans  in  the  full-scale-development  contract  brought  GE  very  close  to  coropleting 
all  tasks  req’iired  for  initial  production  release.  Using  a  modification  in  the  full-scale- 
development  contract,  the  Air  Force  initiated  a  continuing  engineering  development  focused 
on  four  “post  qualification”  development  efforts  that  would  yield  operati  nal  capability 
release  under  the  four-step  development  process; 

(a)  “development  and  test  requirements  of  FllO-GE-100  engine  parts  not  included  in 
the  FCA  [functional  configuration  audit]  configuration],] .  .  .  parts  never  qualified 
and  parts  which  have  been  qualified,  but  may  be  manufactured  by  a  second, 
supplemental  vendor;" 

(b)  improvement  and  verification  of  “the  predicted  life  of  life-limiting  parts  of  the 
FllO-GE-100  engine,  thus  permitting  the  achievement  of  the  4,000  TAG  cycle 
(AMT  VII)  inspection  interval  goal;’ 

(c)  “simulated  altitude  testing  to  characterize  engine  performance  and  deterioration, 
to  update  the  computer  model  and  to  define  control  schedule  tolerances;"  and 

'®The  production  run  could  be  much  Irrgcr.  One  estimate  at  the  time  expected  the  Air  Force  to 
buy  1,200  engines  during  FYs  85-87  and  1,5  »0  in  FY's  88-90  for  the  F-15  and  F-16,  and  the  Navy  to 
buy  600  engines  for  the  F-14  during  FYs  87-;i0  Significant  foreign  military  sales  were  also  expected. 
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(d)  “definition  and  qualification  of  selected  improvements  and  modifications  to 
problems  identified  during  . . .  testing.* 

That  effort  effectively  extended  the  fuII-scale-development  contract  until  December  1985  and 
added  $22.3  million  to  the  contract.'^ 

Multinational  coproduction  of  F-16  subsystems  made  point  (a)  important.  The  F-16 
was  seen  as  a  multinational  fighter,  and  the  identification  of  the  F-13  as  the  first  application 
for  the  FI  10  opened  the  wuy  for  more  detailed  selections  of  manufacturing  vendors  for  engine 
subsystems.  The  Air  Force’s  new  doctrine  on  extending  engine  lives  added  point  (b)  by 
effectively  requiring  that  all  cold-section  components  have  a  life  of  8,000  cycles,  twice  that  of 
the  hot  section  demonstrated  in  the  full-scale  development.  Persistent  “screech*  and 
“thump,"  and  other  p-  oblems  discovered  late  in  development  motivated  point  (d).  Screech 
was  a  dangerous  condition  thnt  resulted  from  dynamic  pressure  oscillation  in  the  augmentor 
under  certair.  predictable  flight  circumstances;  thump  resulted  from  a  shock  wave  in  the 
engine  inlet  that  was  not  dangerous,  but  that  disturbed  enough  pilots  for  the  Air  Force  to 
seek  a  solution.  Section  6  returns  to  these  problems. 

Like  FSD,  this  effort  ran  fairly  smoothly.  The  inspection  interval  for  cold  sections  of 
the  engine  was  successfully  extended.  The  sources  of  the  scr  eech  and  thump  problems  were 
determined,  and  an  interim  fix  was  developed  for  screech.  The  Air  Force  began  to  flight-test 
a  large-mouth  inlet  on  the  F-16  that  allowed  the  aircraft  to  take  greater  advantage  of  the 
engine’s  thrust  potential  and,  coincidentally,  eliminated  the  tliump  problem.  Problems  with 
qualifying  the  engine  monitoring  system,  on  the  other  hand,  persisted.  The  Air  Force  and 
GE  negotiated  a  number  of  smal’  changes  in  work  scope.  Tire  largest  was  a  (not  to  exceed)  $5 
million  adjustment  in  the  contract  price  to  pay  for  additional  work  on  F110/F-15E  integration 
in  May  1986.  Development  activities  continued,  but  the  focus  of  the  FllO  program  shifted 
rapidly  to  production  as  the  Air  Force  accepted  the  first  production  engine  in  February  1986, 
accepted  the  first  operational  engine  in  October  1985,  and  fielded  the  first  squadron  of  F- 16s 
equipped  with  FllO  engines  on  schedule  in  late  1986. 

Further  Testing  and  Development 

As  F-16s  designed  to  use  the  FllO  became  available.,  the  Air  Force  implemented  a 
Lead  t’ne  Force  program  called  Pacer  Ten  to  monitor  the  engines’  performance  in  the  field 

•flU.S.  Air  Force,  Aeronautical  Syctems  Division,  Modification  P00039  to  Contract  82-C-0257, 
Wright-Patterson  Air  Force  Base,  Ohio,  26  July  1985,  definilijed  by  Modification  P00045  to  Contract 
82-C-0257.  p.  1;  Attachment  1,  pp.  1-4. 
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and  prepare  to  respond  to  operational  problems  detected.  That  program  provided  for  ein 
accelerated  usage  rate  on  early  engines  delivered  to  the  field  and  special  monitoring  to  look 
for  problems  that  might  appear  in  the  fleet  as  engine  flying  hours  accumulated.  And  funding 
for  the  Component  Improvement  Program,  conducted  jointly  for  the  FlOl  and  FllO  engines 
to  reflect  the  close  relationship,  provided  a  way  for  the  Air  Force  to  develop  solutions  for 
problems  that  arose  among  fielded  engines.  Congressional  decisions  in  late  1986  to  reduce 
funding  for  the  CIP  and  encourage  contractors  to  assume  more  responsibility  for  the  cost  of 
engine  modifications  promised  to  complicate  the  continuing  development  and  refinement  of 
the  FllO  engine.*'^ 

The  Expenditure  Profile 

Figure  4.1  summarizes  the  total  Air  Force  expenditure  profile  for  that  development. 
G£  put  additional  resources  into  the  continuing  development,  the  most  important  being  at 
least  $20  million  during  its  development  of  the  FIOIX  demonstrator.  The  total  cost  of  $330 
million  (in  then-year  dollars)  was  substantially  less  than  the  $1.5  billion  that  the  Air  Force 
would  have  expected  to  pay  to  develop  a  new  “centerline"'  engine — an  engine  based  on  a 
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Figure  4.1 — Air  Force  Expenditure  ir  FllO-GE  100  and  FlOO-PW-220  Developments 


20Aldridge,  1986,  p.  219. 
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new  design  instead  of  a  derivative  design.^^  In  the  end,  relative  to  the  F100{3)  engine  used 
by  the  FllO  development  as  a  baseline  for  comparison,  the  new  engine  essentially  eliminated 
stall-stagnations,  instability  in  the  augmentor.  and  trim  during  maintenance;^^  improved 
pump  reliability  an  order  of  magnitude;  halved  resources  required  for  intermediate 
maintenance;  more  than  doubled  the  time  between  depot  overhauls;  and  substantially 
reduced  performance  loss  over  the  life  of  the  engine  (see  Table  4.2).  The  basic  improvements 
realized  actually  exceeded  those  expected  when  the  development  began.  It  became  a 
successful  competitor  not  just  for  Air  Force  purchases  but  also  for  foreign  military  sales, 
where  its  thrust  advantage  over  the  FlOO-PW-220  gave  it  additional  appeal. 

THE  P&W  F100-PW-220  AND  RELATED  FIDO  ENGINES 

In  comparison  with  the  development  of  the  FllO-GE-100,  that  of  the  FlOO-PW-220 
looks  much  less  focused  (see  Table  4.1).  But  that  development  occurred  in  a  different 
context.  In  the  late  1970s,  the  FlOO  was  the  Air  Force’s  most  important  engine,  both  because 
of  its  prominence  in  the  fleet  and  its  continuing  high-level  production  rate  and  because  it  was 
creating  such  difficult  problems  for  the  Air  Force.  The  Air  Force  and  its  manufucturer,  P&W, 

Table  4J2 

Performance  Improvementa  Allowed  by  the  FllO-GE-lOO  and  F100'PW>220 


Characteristic 

F100(3) 

FllO-GE-lOO 

I''100.PW.220 

Operability/Safety 

Stall-stagnationa/lOOO  FH 

0.2-1 

0 

0-0.05 

Pump  reliability/ 

100,000  FH 

0.9 

0.05-0.1 

0.1-0.2 

Augmentor  operation 

R2,  R3  instability 

no  instability 

no  instability 

Supportability 

Trim  time  (minutes) 

80 

none 

none 

CosUFH 

$585 

$215-315 

$285-345 

MMHTH 

8.5 

3.3-4.3 

4.0-4.6 

Removals/lOOO  FH 

7.3 

3.0-^.0 

3.6-4.4 

Life  of  Hot  Section  (cycles) 

1,800 

4,000 

4,000 

Performance  Loss 

19^/400  cycles 

<2%/4,000  cycles 

<2%/4,000  cycles 

SOURCE:  Bethel  and  V/ood,  1986. 

NOTES:  R2  and  R3  are  the  namca  of  spray  rings  in  the  augmentor.  Trim  is  a  time-intensive  maintenance 
activity  analogous  to  tuning  an  engine.  FH  is  flight  hours.  MMH  is  maintenance  man-hours. 


2*Ktnnedy,  1985,  p.  1. 

22Trim  is  a  time-intensive  activity  analogous  to  tuning  an  engine;  it  could  be  eliminated  when 
the  FllO-GE-100  offered  a  closed-loop  control  system. 
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were  actively  atu«''''ing  those  problems  on  a  number  of  fronts.  Efforts  on  five  engine 
subsystems  are  of  w':ular  interest  to  us; 

*  increased  life  core  (ILC) 

*  improved  augmentor  (including  a  light.ofr  detector,  LOD,  end  a  dual-ignition 
system,  DIS) 

*  digital  electronic  engine  control  (DEEC) 

*  gear-type  fuel  pump  and  bypass  valve  (GP,  BPV) 

*  engine  monitoring  system  (EMS). 

P&W  ultimately  incorporated  all  five  new  systems,  together  with  some  less  important  ones, 
in  the  FlOO-PW-220  engine  it  offered  for  competition  with  the  FllO-GE-100  in  the  Alternate 
Fighter  Engine  competition.  For  all  intents  and  purposes,  however,  the  -220  development 
was  just  one  of  a  series  of  closely  related  efforts  that  P&W  pursued  to  enhance  its  FlOO 
family  of  engines.  P&W  incorporated  each  of  the  above  changes  either  in  new  production 
FlOO  engines  or,  thror  ;h  retrofit,  in  existing  FlOO  engines  in  the  fleet.  Because  all  such 
development  efforts  occurred  in  tandem,  it  is  hard  to  disentangle  them.  In  fact,  the 
development  program  for  each  modification  created  information  that  supported  the  ultimate 
creation  of  the  *220  engine.  In  the  end,  however,  a  single  office  in  the  Propulsion  SPO  had 
responsibility  for  .he  -220  development  itself  and  ensured  that  it  occurred  in  an  orderly, 
efficacious  manner. 

On  the  basis  of  a  series  of  business  strategy  meetings  and  an  evaluation  of  an 
unsolicited  FlOO  growth  proposal  from  P&W,  the  Air  Force  initiated  an  EMDP  for  the  P&W 
FlOO  at  the  same  time  that  it  began  the  FlOl  DFE  EMDP.  The  FVopulsion  SPO  saw  this 
program  as  part  of  a  broader  effort  to  improve  the  FlOO,  motivated  at  least  as  much  by  the 
opportunity  to  use  the  EMDP  funding  vehicle  as  by  specific  interest  in  an  engine  competition 
at  the  time.  While  the  EMDP  would  address  operability  problems,  the  CIP  would  address 
durability,  and  P&W's  independent  research  and  development  (IR&D)  program  would 
address  enhanced  thrust.  The  discussion  below  focuses  on  the  EMDP,  then  the  CIP,  and 
finally  the  full-scale-development  program  that  yielded  the  FlOO-PW-220  and  related 
innovations. 

The  Engine  Model  Derivative  Program 

The  flnn-fixed-price  FlOO  EMDP  contract  called  for  P&W  to  “conduct  a  pre-FSD 
limited  development/validation  uctirity  which  introduces  demonstrated  advanced  component 
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technology  into  the  FlOO  production  engine  design  to  provide  the  basis  for  capitalizing  on 
improvements  in  specification  performance  and  lowering  ownership  costs."  The  work 
involved  a  "limited  development  and  flight  test  program  required  to  obtain  data  defining  the 
operational  benefits  and  capability  of  a  PlOO  engine  in  incorporaUing]  a  Digital  Electronic 
Engine  Control  (DEEC)  System,  a  Swirl  Augmentor  (SA),  an  augmentor  Light-Off  Detector 
(LOD),  and  augmentor  Dual  Ignition  System  (DIS)."  To  do  so,  P&W  was  to  complete  initial 
design  of  the  LOD  and  DIS  and  refine  other  designs  as  necessary,  procure  or  refurbish 
hardware  for  two  FlOO  EMDP  flight-test  kits,  conduct  a  specified  set  of  ground  tests,  and 
support  up  tc>  19  F-15  and  53  F-16  test  flights.  This  effort  was  priced  at  $19.6  million  and 
scheduled  to  continue  through  September  1983.^^  The  Navy  and  NASA  pursued  related 
interests,  adding  their  own  funding  and  engines  to  the  research  effort. 

Th.e  development  got  off  to  a  rocky  start.  A  lack  of  funding  e irly  in  the  contract 
delayed  the  start  of  testing  at  Arnold  Engine  Development  Center  (AEDC),  the  Air  Force’s 
principal  engine  ground-test  facility,  from  July  1979  to  March  1980.  'The  delay  threw  off  the 
test  schedule  at  AEDC  and  complicated  P&W’s  effort  to  get  access  when  funding  did  become 
available.  Once  testing  began,  P&W  quickly  established  that  the  swirl  augmentor,  which  it 
had  proposed  to  use  as  a  proven  technology  in  the  EMDP  configuration,  would  not  enhance 
performance  as  expected;  Section  5  addresses  this  problem  in  more  detail.  The  development 
then  settled  down,  and  the  program  generated  results  as  expected. 

'The  work  scope  for  this  effort  expanded  dramatically  in  1981.  Early  plans  for  the 
E!MDP  anticipated  program  grrowth  to  cover  a  broad  range  of  engine  improvements.  Perhaps 
growing  Air  Force  interest  in  competition  at  the  time  also  encouraged  growth  in  the  FlOO 
E'lMDP.  In  June  1981,  the  Air  Force  added  $11.2  million  to  the  contract  price  to  pay  for 
additional  ground  tests,  which  included  selective  testing  of  an  expanded  engine 
configuration.  The  new  configuration  added  an  increased  airflow  fan,  compressor  durability 
and  performance  improvements,  and  an  increased-durability  hot  section. Strong 
Congressional  support  led  to  additional  modifications  in  August  1982  and  July  1983  that 
changed  the  tests  required  by  the  original  plan  and  substantially  increased  the  work  scope  to 
address  a  new  long-life,  low-pressure  turbine;  design  changes  in  the  anti-icing  and  fuel  pump 
hardware;  and  substantially  more  sea-level,  altitude,  and  AMT  testing  of  the  expanded  FlOO 

23U.S.  Air  Force,  Aeronautical  Systems  Command,  Letter  Contract  22657-79-C-0541,  Wright- 
Patterson  Air  Force  Base,  Ohio,  1  July  1979;  definitized  in  Modification  PZOOOl  to  79-C-0541,  1  May 
1980.  Attachment  1,  pp.  1-8. 

i^U.S.  Air  Force,  Aeronautical  Systems  Command,  Modification  P00008  to  Contract  79-C-0541, 
Wright-Patterson  Air  Force  Base,  Ohio,  8  June  1981;  definitized  in  Modification  POOOlO  to  Contract  79- 
C-0541,  5  November  1981. 
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EMDP  configuration.  Those  changes  added  a  total  of  $51.4  million  to  the  contract  price  and 
extended  the  contract  to  August  1986.^^  Additional  changes  continued  to  expand  this 
contract.  Only  the  early  portion  of  work  done  under  this  program  fed  into  the  FlOO-PW-220, 
however.  Additional  work  would  lead  to  the  next  generation  of  derivative  P&W  engines,  the 
FlOO-PW-229  improved  performance  engine  (IPE). 

The  Componont  Improvamant  Program 

At  the  same  time,  the  Air  Force  was  using  its  FIDO  CIP  to  increase,  among  other 
things,  the  durability  of  the  FlOO  engine  core  and  thereby  reduce  operating  costs.  The  CIP 
used  a  standard  cost-plus-award-fee  (CPAF)  contract  to  cover  this  work  and  all  the  other 
tasks  included  in  the  CIP  work  scope  at  that  time.^^ 

This  effort  represented  the  first  formal  application  of  the  Air  Force’s  new  ENSIP 
framework  for  risk  assessment.  Progress  on  the  life  of  the  engine  core  continued  over  a  long 
period  as  the  time  between  depot  overhauls  rose  from  1,250  cycles  in  1979  to  4,000  cycles  in 
the  FlOO-PW-220.  Progress  made  during  1981-1985  has  been  directly  associated  with  the 
effort  to  develop  the  increased  life  core  used  in  the  FlOO-PW-220.*’  The  CIP  yielded  a  stand¬ 
alone  upgrade  package  that  could  retrofit  existing  FlOO  engines  with  a  4,000-cycle  core  by 
late  1985.  This  same  effort  yielded  the  increased-life  core  required  in  the  FlOO-PW-220. 
During  this  time,  test  engines  passed  frequently  among  the  CIP,  EMDP,  and  full-scale- 
development  programs,  illustrating  the  coordination  among  these  efforts.  And  the  CIP 
program  played  an  active  role  in  final  product  verification  of  the  F100-PV/-220,  running 
endurance  and  other  ground  tests.  In  the  closing  months  of  the  product-verification  effort, 
-220  testing  was  the  most  important  effort  in  the  FlOO  CIP.  CIP  testing  continued  following 
product  verification,  in  preparation  for  the  normal  role  of  CIP  of  identifying  and  resolving 
problems  that  arise  in  the  course  of  normal  engine  operations. 

During  the  1981-1985  period,  budgetary  delays  frequently  led  to  the  use  of  continuing 
resolution  authorization  to  fund  the  CIP.  Because  the  actual  tasks  undertaken  in  the  CIP 
are  extremely  responsive  to  the  level  of  funding  available  in  any  year,  such  delays  created 
substantial  uncertainty  and  confusion  in  the  CIP.  Apparently  because  CIP  efforts  related  to 

25U.S.  Air  Force,  Aeronautical  Systems  Command,  Modification  P00012  to  Contract  79-C-0541, 
13  August  1983,  and  Modification  P00014  to  Contract  79-C-0641, 1  July  1984,  Wright-Pattarson  Air 
Force  Base,  Ohio;  definitized  in  Modification  P00015  to  Contract  79-C-0541,  31  July  1984. 

26By  1980,  the  CIP  concept  was  fairly  mature  and  the  contractual  vehicle  used  to  implement  it 
each  year  was  fairly  standard.  Some  CEP  flight-test  activities  were  conducted  under  firm-fixed-price 
level-of-eflbrt  terms  before  1980.  But  CPAF  terms  dominated  the  CIP  during  the  years  of  greatest 
interest  to  us.  The  CIP  normally  uses  such  terms. 

27See  U.S.  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion,  n.  d.;  Nelson,  1083b. 


the  increased'Iife  core  and  the  F100-PW>220  itself  had  high  priority,  this  uncertainty  did  not 
seriously  slow  these  efforts. 

Full-Scale  Development 

The  Air  Force  initiated  the  full-scale-development  program  relevant  to  the  FIOO-PW- 
220  in  September  1982.  The  full-scale-development  program  of  interest  to  us  initially  sought 
to  qualify  three  engine  improvements  for  production  incorporation,  none  of  them  the  FIOO- 
PW-220  itself,  but  each  closely  related  to  it; 

“(1)  A  digital  Electronic  Engine  Control  System/Gear-Type  Main  Fuel  Pump 
(DEEC/Pump)  System  for  FlOO-PW-200  engine  production  availability  in 
March,  1984; 

(2)  A  Gear-Type  Main  Fuel  Pump/Bypass  Valve  (PumpA^'alve)  System  for 
FlOO-PW-200  engine  production  availability  and  field  retrofit  availability 
in  July,  1984;  and 

(3)  An  Engine  Monitoring  System  (EMS)  for  use  in  conjunction  with  the 
DEEC  System  for  engine  production  availability  and  field  retrofit 
availability  in  July,  1985; 

which  are  intended  to  improve  durability,  maintainability,  operability, 
reliability,  safety  and  life  cycle  cost  of  the  F-16/F100  Weapon  System." 

This  firm-fixed-price  effort,  priced  at  $65.4  million,  would  design  the  four  electronic  units 
that  made  up  tho  EMS,  complete  the  design  of  the  gear  pump  and  the  DEEC/Pump  system, 
and  modify  the  engine  case  design  to  mount  new  components.  It  would  fabricate  new 
hardware  for  the  bypass  valve  and  modify  existing  engine  hardware  to  accept  the  DEEC. 

And  it  would  conduct  extensive  ground  tests  and  F-16  flight  tests.^^  The  effort  set  clear 
performance  goals  for  selected  parts  of  the  development  (see  Table  4.3  for  an  example  from 
the  DEEC/gear  pump  portion  of  the  FlOO  development);  they  would  provide  a  basis  for  goals 
in  the  FlOO-PW-220  program. 

The  FSD  program  proceeded  smoothly  into  1983,  exercising  some  important  options. 
The  program  finally  developed  a  specification  for  the  FlOO-PW-220  engine  in  June  1983. 
Recall  that  the  Air  Force  issued  its  final  request  for  proposals  for  the  Alternate  Fighter 
Engine  production  competition  in  May  1983.  This  new  specification  brought  together  the 
DEEC,  gear-type  fud  pump,  and  engine-monitoring  system  envisioned  in  the  initial  work 
scope  for  this  contract ,  and  the  increased-life  core  that  P&W  had  been  developing  in  the 

*®U.S.  Air  Force,  Aeronautical  Systems  Command,  Contract  F33657-82-C-2199.  Wright- 
Patterson  Air  Force  Base.  Ohio,  30  September  1982,  Attachment  1,  pp.  1-2. 


Table  4.3 

Improvement  Goals  for  the  FlOO  DEEC/Gear  Pump  Development 


-46- 


levelopment;  KIAS  is  knots-indicated  speed;  MFP  is  main  fuel  pump;  SV  is  shop  visit 
®Dt>Ps  not  include  diopnostics. 

^0  be  demonstrated  during  this  development  program  with  primary  fuel  only. 
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FlOO  CIP,  to  create  an  engine  that  could  satisfactorily  compete  with  the  FllO-GE-100.^ 
Initially,  it  was  *'simply”  referred  to  as  the  FIDO  DEEC/GP/EMS/ILC,  emphasizing  that  the 
developers  viewed  it  at  the  time  as  just  one  more  member  of  an  expanding  FIDO  family.  It 
can  be  thought  of  as  an  enhancement  that  incorporates  all  of  the  changes  discussed  here  in  a 
single  block  change.  The  FlOO-PW-220  specification  was  completed  later  in  1983,  and  final 
testing  of  the  configuration  began  shortly  thereafter.  The  DSARC  accepted  the  Air  Force’s 
plans  for  dealing  with  remaining  risk  and  the  Alternate  Fighter  Engine  source  selection  in 
February  1984  awarded  P&W  41  -220  engines  for  F-15s,  a  quarter  of  the  total  number  of 
engines  that  the  Air  Force  purchased  in  the  first  year  of  the  competition.^^  This  led  quickly 
to  additional  buys  by  Egypt,  Korea,  Thailand,  and  Singapore;  other  foreign  buyers  joined 
them  later. 

Shortly  before  then,  in  September  1983,  the  full-scale-development  program  expanded 
to  include  an  F-15  application  of  the  DEEC/pump  system  by  December  1985,  adding  $5.7 
million  to  the  contract  price. In  November  1985,  the  Air  Force  officially  qualified  the  FIOO- 
PW-220  engine  for  application  in  the  F-15  and  F-16.  The  Air  Force  accepted  its  first  -220 
engine  in  tlie  same  uionih.  The  first  engine  reached  the  field  in  an  operational  aircraft  in 
August  1986,  six  months  ahead  of  schedule.  As  noted  above,  CIP  shifted  its  emphasis  from 
supporting  the  qualification  effort  to  supporting  the  -220  in  the  field.  Anticipated  shortfalls 
in  CIP  funding,  however,  were  expected  to  hamper  this  effort. 

In  the  meantime,  development  continued  on  the  -220.  Application  to  the  F-15E 
continued.  The  Air  Force  added  $4.2  million  to  the  contract  price  in  August  1955  to  redesign 
components  of  the  FlOO-PW-220  EMS  and  develop  software  and  maintenance  procedures  for 
the  -220  EMS  by  February  1987. It  added  emother  $1.9  million  as  recently  as  1990  for 
additional  work  on  the  EMS.^^ 

As  the  scope  of  the  full-scale-development  contract  makes  clear,  the  developers  were 
not  wrong  to  think  of  the  FlOO-PW-220,  a.k.a.  the  FlOO  DEEC/GP/EMS/ILC,  as  only  one 

28U.S.  Air  Force,  Aeronautical  Systems  Command,  Modification  FD0005  to  Contract  82-C-2199, 
Wright-Pattcrson  Air  Force  Base,  Ohio,  29  June  1983. 

effect,  P&W  produced  engines  for  the  F-15  during  the  fu  st  year  of  the  production  contract 
while  GE  produced  engines  for  the  F-16.  As  time  has  passed,  P&W’s  share  of  new  production  contracts 
has  increased,  enabling  it  to  supply  engines  for  F-16s  as  well. 

^*U.S.  Air  Force,  Aeronautical  Systems  Command,  Modification  P00006  to  Contract  82-C-2199, 
Wright-Patterson  Air  Force  Base,  Ohio,  30  September  1983;  definitized  by  Modification  P00009  to 
Contract  82-C-2199,  31  May  1984. 

02U.S.  Air  Force,  Aeronautical  Systems  Command,  Modification  P00012  to  Contract  82-C-2199, 
Wright-Patterson  Air  Force  Bcise,  Ohio,  30  September  1983;  definitized  by  Modification  P00014  to 
Contract  82-C-2199,  27  February  1987. 

03U.S.  Air  Force,  Aeronautical  Systems  Command,  Modifications  P00017,  P00018,  and  P00020 
to  Contract  82-C-2199,  Wright-Patterson  Air  Force  Base,  Ohio,  11  May  1990,  6  September  1990,  and  10 
October  1990,  respectively. 
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among  several  related  efforts.  Other  FIDO  development  efforts  continued  during  this  period. 
The  gear  pump/bypass  valve  effort  qualified  a  new  production  kit  by  July  1984  and  a  retrofit 
kit  a  month  later,  beginning  the  most  complex,  comprehensive  retrofit  effort  to  date  in  the 
FIDO  program.  Following  its  introduction,  the  Air  Force  ran  a  one-year  field  service 
evaluation  of  the  retrofit.  Eight  modified  engines  were  flown  on  an  intensified  operational 
schedule  and  carefully  monitored  for  problems.  Few  appeared,  and  none  was  serious.  By 
May  1985,  P&W  had  qualified  the  DEEC/gear  pump  modification,  allowing  the  Air  Force  to 
initiate  a  similar  two-year  field  service  evaluation  of  41  production  engines.  It  too  found  no 
serious  difficulties.  By  late  1985,  the  Air  Force  was  using  the  CIP  to  run  yet  another  field 
service  evaluation  for  an  FlOO-PW-200  DEEC/pump/ILC  retrofit,  with  equally  good  success. 
And  P&W  and  the  Air  Force  had  reached  agreement  on  an  extensive  program  to  retrofit 
existing  FlOO-PW-lOO  and  -200  engines  with  a  4,000-cycle  core  upgrade.  All  these  efforts 
produced  data  very  likely  to  assist  developers  as  they  approached  the  final  qualification  of 
the  FlOO-PW-220. 

The  Expenditure  Profile 

Such  joint  testing  makes  it  difficult  to  attribute  all  costs  to  one  program  or  another. 
The  Air  Force  relates  the  costs  shown  in  Figure  4.1  with  the  -220  development.^*  As  with  the 
FllO  development,  the  -220  development  cost  the  Air  Force  substantially  less  than 
development  of  a  new  centerline  engine  would  have  cost — $227  million  rather  than  $1.5 
billion.  Relative  to  the  F100{3)  engine  used  by  the  Air  Force  as  a  baseline  for  comparison, 
the  FlOO-PW-220  almost  eliminates  stall-stagnations,  eliminates  augmentor  instability  and 
trim,  increases  pump  reliability  five-  to  ninefold,  reduces  the  resources  required  for 
intermediate  maintenance  40  to  50  percent,  doubles  the  time  between  depot  overhauls,  and 
significantly  reduces  performance  loss  over  the  life  of  the  engine  (see  Table  4.3),  The  realized 
basic  improvements  actually  exceeded  those  expected  when  the  development  began.  The 
-220  engine  continued  to  compete  successfully  with  the  GE  FllO  for  Air  Force  production 
contracts  until  the  IPE  engines  replaced  both  engines  in  the  continuing  GE-P&W  competition 
in  1991. 


These  costs  include  funds  from  program  elements  27133F  (gear  pump),  64209F  (FlOO 
durability),  64223F  (FlOO-PW-220  FSD),  64268F  (ILC).  See  ASD  Propulsion  SPO,  Office  of  Program 
Control  (YZPR),  Report  to  ASD  History  Office  for  January-June  1986,  Wright-Patterson  Air  Force 
Base,  Ohio,  n.d. 
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BnOAOER  CONTEXT 

Within  the  Air  Force,  the  Propulsion  SPO  oversaw  the  development  and  subsequent 
production  of  the  two  derivative  engines.  Unlike  most  SPOs,  the  Propulsion  SPO  oversees  a 
number  of  systems  and  in  fact  remains  intact  as  one  system  starts  up  and  another  closes 
down.  During  tho  development  of  the  FI  30  and  -220  engines,  the  Propulsion  SPO  oversaw 
RDT&E  activities  for  a  fairly  large  number  of  engine  programs.  Figure  4.2  illustrates  the 
position  of  the  two  engines  we  are  studying  relative  to  the  other  programs  overseen  by  the 
Propulsion  SPO  at  the  time.  The  figure  shades  relevant  funds  from  the  FlOl  and  FlOO 
EMDPs,  which  appear  early  in  the  overall  EMDf ,  and  the  Alternate  Fighter  Engine  (AFE) 
program,  which  effectively  paid  for  full-scale  development.^  As  the  figure  indicates,  these 
engines  accounted  for  u  small  fraction  uf  the  SPCs  RDT&sE  activities  over  a  period  of  about 
six  years. 

Two  other  points  about  this  figure  are  worth  emphasizing.  First,  the  FllO-GE-100 
and  FlOO-PW-220  engines  are  part  of  a  continuum.  They  grew  out  of  established  engine 


SOURCE:  U  S.  Air  Force,  Aeronautical  Systems  Division  Deputy  lor  Propulsion.  1986,  p.  S-A-l  l . 


Figure  4.2 — Air  Force  RDT&E  Expenditures  on  Engine  Programs 


35As  noted  above,  the  CIP  accounted  for  a  portion  of  the  -220  development  and  continued  a  low 
level  of  expenditure  for  both  programs  following  FSD.  But  these  quantities  are  not  large  enough  to 
change  the  general  picture  offered  by  Figure  4.2. 
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programs  (the  FlOO  and  FlOl)  and  provided  the  basis  for  follow-on,  derivative  engine 
programs  (the  improved  performance  engines,  or  IPEs).  This  explicit  derivative  development 
strategy  is  a  reflection  of  a  more  general  principle  in  engine  development.  Even  new 
centerline  engines  inherit  a  great  deal  of  technology  from  earlier  engines.  The  management 
continuity  offered  by  using  a  single  SPO  to  oversee  all  engine  developments  facilitates 
transfer  of  technology  from  one  engine  to  another,  transfer  of  lessons  learned  in  managing 
one  engine  development  to  the  next  engine  development,  and  transfer  of  lessons  learned  from 
working  with  a  particular  contractor  from  one  engine  contract  to  another.  (We  return  to 
these  themes  in  Section  5.)  Both  the  FllO-GE-100  and  FlOO-PW-220  developments  were 
strongly  shaped  by  earlier  technological,  management,  and  operational  experience  with  the 
FlOO-PW-lOO  in  particular. 

Second,  at  any  point  in  time,  engine  developments  do  not  occur  in  isolation.  Clearly, 
the  competition  between  the  FllO-GE-100  and  FlOO-PW-220  developments  was  apparent  in 
both  developments  from  beginning  to  end.  It,  more  than  any  other  factor,  is  what  obse~v’''r3 
associate  with  these  engines.  In  addition,  the  FlOO-PW-220  was  only  one  of  several  closely 
related  developments  making  incremental  improvements  in  the  FlOO;  RDT&E  expenditures 
for  such  improvements  and  for  development  to  go  beyond  the  FlOO-PW-220  are  captured  in 
the  FlOO  and  EMDP  areas.  The  FI  10  and  FlOl  developments  were  also  working  with  very 
similar  engines;  however,  unlike  the  improved  FlOO  developments,  they  saw  little 
coordination  or  synergy.  The  tight  schedule  imposed  by  Congress  on  the  B-IB  development 
led  the  Propulsion  SPO  to  isolate  its  FlOl-GE-102  development  to  minimize  outside  factors 
that  might  complicate  the  development.  In  this  case,  the  Propulsion  SPO  effectively  had  to 
forgo  sm  opportunity  for  the  kind  of  synergy  it  exploited  'n  its  FlOO  improvements. 

Figure  4.3  offers  another  way  to  look  at  the  FlOO-GE-100  and  FlOO-PW-220  programs 
in  relation  to  other  Propulsiot;  SPO  activities.  The  percentage  of  entries  in  the  Propulsion 
SPO  Commander's  “Weekly  Activity  Report"  relevant  to  a  particular  program  appears  on  the 
ordinate.  If  we  take  the  percentage  of  citations  as  a  measure  of  the  Propulsion  SPO 
commander’s  interest  in  a  certain  program,^®  we  see  interest  growing  with  the  expenditure 
on  the  FllO-GE-100  and  FlOO-PW-220  programs,  but  reaching  a  higher  proportion  of  the 
commander’s  focus  than  of  his  budget.  In  fact,  the  competition  between  these  engines 

36 Alternatively,  because  the  ASD  commander  is  the  consumer  of  this  weekly  report,  we  might 
take  it  as  a  measure  of  the  ASD  commander’s  interests  in  activities  within  the  Propulsion  SPO.  Either 
way,  the  percentage  offers  a  useful  index  of  Air  Force  management  interest. 
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Figure  4.2— Propulsion  SPO  Management  Focus  During  the  FllO-Gi.  iOO  and 

FlOO-PW-220  Developments 

became  t>ie  single  most  important  iiurm  on  the  commander’s  agenda  dunng  1982-1986^'^  that 
is  reP.ected  in  the  percentages  in  Figure  4.3.  The  fa.t  that  the  SPO  commander  gave  a 
disproportionate  amount  of  attention  to  these  programs  during  their  developments  may  help 
explain  their  success,^* 

SUMMARY 

The  Great  Engine  War  between  the  FllO-GE-100  and  FlOO-PW-220  is  generally 
regarded  as  a  major  success  in  Air  Force  a'-quisition.  A  significarit  part  of  that  success  can  be 
attributed  to  the  quality  of  the  two  engines  and  the  success  of  the  development  programs 
tliat  made  the  Great  Engine  War  possible.  Although  the  development  programs  for  the  two 
engines  are  intricately  intertwined,  the  two  proceeded  quite  differently  and  involved 
.significantly  different  technological  and  management  problems. 

on  interviews  with  former  SPO  staiT. 

38The  commander’s  interest  remained  high  as  these  programs  left  development,  and  RDT&E 
expenditures  fell  as  production  began.  Budget*  and  SPO  staffing  to  oversee  these  pmgi  ame  more 
liiDii  enough  as  production  began  to  increase  total  annual  program  expenditures.  The  commander’s 
interest  did  not  rise  in  proportion  to  this  change;  the  development  phase  attracted  a  diei'roportionate 
share  ot  his  interest. 
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The  two  share  the  feature  of  bein^  derivative  developments,  but  they  were  derivative 
in  different  ways.  The  FllO-GE-100  evolved  systematically  from  the  FIOIX  demonstrator 
that  GE  constructed  to  extend  the  capability  it  had  created  with  the  FlOl  engine  developeH 
for  the  B-1.  The  Air  Force  used  the  FlOlX  as  the  basis  for  a  low-risk  demonstration  and 
validation  program  that  required  only  marginally  new  design  work;  mainly,  it  called  for  GE 
to  characterize  the  engine  it  had  previously  designed  and  demonstrate  it  in  a  limited  flight- 
test  program.  This  program  began  the  final  refinement  of  that  design,  which  the  formal,  full- 
scale-development  program  then  completed  and  carried  through  product  verification. 

The  FlOO-PW-220  did  not  follow  as  linear  a  path.  P&W  and  the  Air  Force  had  a 
variety  of  programs  under  way  to  introduce  demonstrated  advanced  components  into  the 
FlOO  to  solve  serious  problems  that  the  Air  Force  had  with  operability  and  ownership  cost. 
Each  of  these  developments  involved  marginal  adjustments  in  the  FlOO-PW-lOO  and  -200 
engines  that  dominated  the  Air  Force  fighter  fleet  around  1980.  The  Air  Force  brought  these 
efforts  into  final  engineering  and  product  verification  as  individual  changes  in  the  FlOO 
engine  family,  and  the  FlOO-PW-220  effectively  coalesced  from  those  changes  as  a  composite 
change  that  could  compete  with  the  FllO-GE-100  in  the  Great  Engine  War.  Once  P&W 
finalized  Uie  FlOO-PW-220  configuration,  product  verification  followed  immediately  and 
smoothly.  In  the  end,  both  engines  provided  a  limited  set  of  marginal  adjustments  in 
existing  engines  to  resolve  the  most  serious  problems  facing  the  Air  Force  fighter  fleet  in  the 
late  1970s. 

Although  the  formal  production  competition  of  the  Great  Engine  War  is  inseparable 
from  these  developments,  it  is  worth  keeping  in  mind  that  the  developments  themselves  did 
not  occur  in  a  formal  competition.  In  fact,  while  the  improved  FlOO  developments  were  in 
progress,  P&W  managers  spent  a  significant  portion  of  the  time  attempting  to  prevent  a 
formal  competition.  In  a  formal  development  competition,  we  would  have  expected  much 
more  nearly  equal  treatment  of  the  competitors  and  more  formal  coordination  of  the 
competition  with  common  milestones.  What  we  see  instead  is  a  form  of  competition  more 
familiar  to  the  private  sector,  in  which  two  product  developers  see  tha  potential  for  head-to- 
head  competition  when  they  field  their  products  and  jockey  for  position  as  they  develop  the 
products  they  expect  to  compete.  With  an  extensive  FlOO  improvement  program  in  place  to 
support  the  existing  Air  Force  fighter  fleet,  the  Air  Force  promoted  this  kind  of  competition 
by  initially  focusing  more  resources  on  the  weaker  contender,  GE.  Only  when  GE  had 
demonstrated  that  it  had  a  viable  product  did  the  Air  Force  begin  to  fund  the  two 
developments  at  similar  levels.  But  even  then,  each  development  continued  to  proceed  in  a 
very  different  way. 
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Both  the  Air  Force  and  its  contractors  maintained  freat  flexibility  during  the 
development  of  the  two  engines.  Although  these  engines  had  greater  high-level  visibility 
than  we  might  expect  engines  to  get  during  development,  high-level  officials  did  not  play  a 
significant  role  in  the  developments  other  than  to  assure  funding  to  allow  them  to  continue. 
We  would  not  expect  a  derivative  development  of  a  subsystem  with  no  immediate  combat 
function  to  divert  thoir  attention  from  riskier  developments  more  nearly  focuced  on  a  combat 
mission.  To  the  extent  tliat  high-level  officials  noticed  the  developments,  it  was  because  of 
their  novelty — the  development  of  a  new  engine  to  support  competition — and  P&W  3  efforts 
to  derail  that  competition  before  it  could  start.  The  DS/iRC  intervened  once;  otherwise,  the 
two  developments  proceeded  mainly  under  the  watchful  eyes  of  concerned  Air  Force 
overseers. 

The  existence  of  the  standing  Propulsion  SPO,  with  a  useful  accumulation  of 
experience  in  developing  engines  uven  in  the  early  days,  when  these  developments  began, 
maintained  an  appropriate  balance  of  flexibility  and  control  in  the  absence  of  high-level 
oversight.  The  SPO  repeatedly  found  creative  ways  to  assemble  funding,  maintain  support 
for  continuing  two  parallel  developments,  coordinate  related  and  complex  developments,  and 
react  to  unexpected  outcomes.  This  body  of  experienced  managers  maintained  a  form  of 
sophisticated  control  that  a  SPO  of  the  same  size,  assembled  only  for  the  purpose  of  these 
developments,  probably  could  not  have  provided.®* 

The  dominant  contract  type  in  both  developments  was  a  firm-fixed-price  contract. 
Although  such  contracts  have  been  criticized  as  inappropriate  for  development  programs, 
ihey  appear  to  have  worked  well  for  these  two  developments.  Neither  the  Air  Force  nor  the 
contractors  were  surprised  by  actual  development  costs  and  final  delivery  dates,  and 
performance  levels  were  met  as  expected  when  the  developments  began.  The  fact  that 
derivative  developments  limited  the  risk  in  these  ^jrograms  helps  explain  part  of  the 
outcome.  Many  close  observers  and  participants  also  credit  the  presence  of  competition  and 
the  experience  of  personnel  in  the  SPO. 

This  brief  overview  of  the  two  developments  can  only  hint  at  the  details  relevant  to 
these  summary  remarks.  The  next  section  returns  to  details  of  risk  management. 

3®Looking  across  several  cases,  Glennan,  forthcoming,  notes  that  greater  oversight  does  appear 
to  be  associated  with  higher  occurrence  of  difficulties  in  a  development.  In  part,  that  is  because 
inherently  more  difficult  developments  receive  more  oversight.  It  also  appears  to  be  true  because 
increased  oversight  attenuates  the  flexibility  that  developers  have  to  deal  with  the  difficulties  they 
encounter. 
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S.  RISK  MANAGEMENT  IN  THE  F100-PW-220  AND  F110-GE-100  DEVELOPMENTS 


With  this  basic  history  in  hand,  we  can  now  look  at  a  set  of  specific  topics  in  more 
detail.  This  section  first  looks  at  the  riskc  inherent  in  the  two  engine  developments  and  the 
policies  associated  with  those  risks.  Given  the  risks,  it  then  considers  three  primary  aspects 
of  the  en'.dronment  in  which  the  developments  proceeded  and  asks  how  they  affected  the 
management  of  risk  during  the  developments.  The  first  aspect  is  competition,  the 
characteristic  observers  associate  with  the  developments  first  and  the  dominant  factor  that 
participants  point  to  whon  they  discuss  the  developments.  The  second  aspect  is  the  nat  of 
management  within  the  Propulsion  SPO.  As  a  standing  SPO  not  exclusively  committea 
one  program,  the  Propulsion  SPO  had  experienced  managers  who  could  work  as  a  team  arid 
implement  a  well-articulated  philosophy  of  development.  Perhaps  paradoxically,  the  sense  of 
order  offered  by  this  organization  promoted  a  flexible  management  style  that  allowed 
creative  reaction  to  surprises.  The  third  aspect  we  examine  is  the  contracts  used  to 
implement  the  developments.  Although  contract  type  tends  to  dominate  most  discussions  of 
risk  management,  other  aspects  of  contracting  appear  to  have  played  a  larger  role  in  the 
developments. 

RISKS  INHERENT  IN  THE  DEVELOPMENTS 

Two  factors  tended  to  limit  the  risks  inherent  in  the  FlOO-PW-220  and  FllO-GE-100 
developments.  First,  as  derivative  developments,  they  faced  limited  technological  risk. 
Second,  they  faced  little  external  pressure  to  achieve  unrealistic  schedule,  cost,  or 
performance  goals.  These  factors  limited  the  risks  that  their  developers  had  to  plan  for  and 
hence  simplified  both  developments  significantly. 

Technological  Risk 

Engine  developments  tend  to  be  incremental.  More  than  with  most  major  systems,  we 
can  trace  genealotpes  of  engines  back  through  one  or  more  previous  engines  to  the 
demonstrators  that  first  displayed  the  technologies  embodied  in  the  current  engines. 
"Incrementalism”  is  a  response  to  the  highly  complex  nature  of  engines.  Engine  designers 
and  developers  must  integrate  many  different  engineering  disciplines  to  be  successful.  The 
moving  parts  of  engines  rotate  at  high  speeds.  Their  hot  sections  experience  extremely  high 
temperatures  and  temperature  transients  during  operation.  Air  flow  and  pressure  transients 
within  the  operating  engines  are  extremely  complex.  The  air  pressure  and  temperature  of 
the  environment  in  which  engines  operate  continually  change  in  flight.  And  various  parts  of 
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the  engines  can  have  unanticipated  effects  on  other  parts  as  transients  move  through  the 
engine  in  flight.  High-performance  engines  jointly  optimize  these  elements.  Hence,  unless 
each  of  these  elements  is  working  as  expected,  an  engine  cannot  achieve  the  perfonncUice  euid 
safety  expected  of  it.  As  a  result,  developers  tend  to  make  changes  in  engines  carefully, 
drawing  on  existing  engines  whenever  possible. 

But  incrementalism  can  occur  in  different  ways.  And  in  fact,  engine  developers 
distinguish  two  kinds  of  development — derivative  engines  and  centerline  engines. 

The  engines  we  examine  here  are  derivative  engines,  making  marginal  adjustments  in 
selected  portions  of  the  engines  without  reoptimizing  the  design  of  the  engines  as  a  whole  or 
adding  fundamentally  new  materials  cr  operating  concepts.  Such  changes  allow  engines  to 
evolve.  Engine  developers  see  a  need  for  such  evolution  to  match  the  typical  evolution  of 
aircraft.  Over  the  course  of  a  typical  aircraft  design’s  life,  it  normally  takes  on  new 
subsystems  that  add  weight,  drag,  or  both.  To  maintain  the  performance  originally 
envisioned  for  the  gurcraft  system  as  a  whole,  the  engine  must  keep  pace,  adding  thrust  or 
shedding  weight  to  offset  changes  elsewhere  in  the  aircraft.  Therefore,  a  tradition  of 
derivative  development  of  engines  exists  in  the  engine  community. 

The  developments  we  examine  have  a  slightly  different  goal — to  overcome  basic 
problems  with  operability  and  cost  in  the  FlOO  engine  used  to  power  the  F-15  and  F-16.  But 
the  goal  could  be  achieved  by  similar  means.  GE’s  derivative  approach  responded  directly  to 
the  Air  Force’s  concerns  about  the  FlOO  without  taking  any  large  technological  risks.  P&W’s 
derivative  approach  used  modifications  it  was  developing  to  counter  the  GE  challenge 
without  taking  large  technological  risks.  Both  developments  took  advemtage  of  the  derivative 
tradition  to  make  impressive  imp»-nvements  that  essentially  solved  the  problem  at  hand 
without  taking  large  risks. 

Such  an  evolutionary  approach  is  not  always  sufficient.  For  example,  it  would  not 
have  allowed  the  doubling  in  thrust-to-weight  ratio  achieved  by  the  FlOO  engine.  Nor  would 
such  an  approach  have  yielded  an  engine  that  would  aillow  the  advanced  tactical  fighter 
(ATF)  to  cruise  at  supersonic  speeds  without  lighting  the  afterburner.  Such  revolutionary 
changes  require  the  development  of  a  new  centerline  engine.  In  the  tradition  of  engine 
development,  centerline  engines  draw  heavily  on  existing  technology.  But  they  do  not  allow 
the  low-risk  approach  that  we  saw  in  the  F100-PW-22C  and  FllO-GE-100  developments.  It 
is  significant  that  the  Air  Force  continued  its  derivative  engine  program  while  it  was 
developing  the  engine  for  the  ATF.  The  derivative  engines  that  build  on  those  examined 
here,  the  increased  performance  engines  (IPEs),  addressed  a  set  of  goals  for  improving  thrust 
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in  the  F-16  and  F-16  fleets  that  could  be  achieved  without  the  risk  involved  in  the  ATF 
program. 

Externally  Imposed  Risks 

Two  kinds  of  external  risks  were  important  to  these  developments.  One  was  the  risk 
imposed  by  oversight,  which  can  force  developers  to  compromise  their  own  concerns  about 
risk  minimization  to  achieve  higher  level  goals  imposed  from  above.  The  other  was 
associated  with  competition.  We  discuss  the  first  here  and  save  competition  for  the  next 
subsection,  in  which  we  can  treat  it  in  some  detail. 

These  developments  were  remarkably  firee  of  intervention  firom  above.  They  occurred 
outside  the  normal  DSARC  process  used  to  oversee  large  developments  in  DoD,  because 
engines  rarely  enter  that  process  independently  of  airframes  being  developed  at  the  same 
time.  With  no  airframe  being  developed  to  use  these  engines,  the  DSAKC  entered  their 
development  only  once,  after  full-scale  development  had  begun,  and  required  no  changes  in 
them.  As  derivative  engines,  these  developments  also  had  a  relatively  low  joint  cost  of  about 
$610  million  over  seven  years,  not  enough  to  draw  high-level  interest  unless  they 
experienced  a  major  shortfall  of  some  kind.*  They  did  not. 

In  the  end,  these  developments  did  not  impose  a  major  risk  on  the  Air  Force  or  any 
organization  above  it.  Even  a  substantial  cost  overrun  in  such  a  small  program  would  not 
threaten  any  of  those  organizations.  And  the  Air  Force  was  prepared  to  end  the  FlOl  DFE 
effort  if  cost  got  out  of  hand.  Ending  the  program  would  complicate  the  Air  Force’s 
management  of  the  FlOO,  but  it  would  not  eliminate  any  critical  capability  or  keep  the  Air 
Force  from  addressing  any  important  part  of  its  perceived  threat.  For  similar  reasons,  the 
Air  Force  could  accept  schedule  slips  or  performance  shortfalls  in  either  the  GE  or  P&W 
program,  understanding  that  the  other  was  available  as  a  backup.  We  will  say  more  about 
this  role  of  competition  below.  The  key  point  here  is  that  these  developments  did  not  have 
inherent  features  that  we  would  expect  to  draw  high-level  interest. 

That  is  not  to  say  that  there  was  no  high-level  interest  in  the  development.  In  its 
efforts  to  eliminate  funding  for  GE  and  to  preempt  the  competition  between  GE  and  itself, 
P&W  brought  the  development  to  the  attention  of  a  number  of  congressmen,  senators,  and 
administration  officials.  In  fact,  P&W’s  high-level  efforts  were  unusual  in  engine 
development  and  encouraged  the  Air  Force  itself  to  resist  outside  interference.  In  response 
to  P&W’s  efforts,  congressional  staffs  held  hearings,  began  GAO  investigations,  demanded 
written  documents  from  the  Air  Force,  and  succeeded  in  threatening  the  funding  necessary 

*Thi.s  cost  estimate  is  based  on  the  data  presented  in  Table  3.1. 
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for  competition  to  go  forward.^  These  efforts  brought  the  development  to  the  attention  of 
high-level  Air  Force  ofiScials.  The  Air  Force  responded  by  closing  ranks  at  the  highest  level. 

In  the  end,  higher  level  ofhcials  never  intervened  in  the  details  of  managing  the 
developments.  Once  funding  was  provided,  high-level  officials  withdrew  and  allowed  the 
developments  to  proceed  on  the  terms  set  by  the  Air  Force.  High-level  interest  never  became 
strong  enough  to  override  the  judgments  of  the  managers  directly  responsible  for  the 
developments.  This  basic  fact  significantly  simplified  the  developers’  lives  by  allowing  them 
to  build  slack  into  the  full-scale  developments  that  could  absorb  surprises  that  might  have 
adversely  affected  cost,  schedule,  or  performance.  We  will  say  more  about  this  when  we 
discuss  flexibility  below. 

Summary 

A  derivative  approach  makes  sense  when  it  can  achieve  the  developer’s  goals.  Given 
those  goals,  it  helps  the  developer  minimize  the  risks  associated  with  development.  But  a 
derivative  approach  will  not  work  in  all  applications.  Where  larger  advances  in  technology 
are  desired,  higher  development  risk  is  usually  unavoidable.  Because  these  developments 
were  fairly  small  and  essentially  supported  a  reenginiog  program,  they  did  not  impose  a  high 
risk  on  the  Air  Force  or  any  organization  above  the  Air  Force,  Despite  P&W’s  effort  to  draw 
higher  level  interest  to  the  programs,  such  interest  had  little  effect.  Air  Force  managers 
were  fairly  free  to  structure  the  developments  as  they  saw  fit. 

COMPETITION 

The  presence  of  competition  profoundly  affected  the  environment  in  which  the  Air 
Force  and  its  contractors  managed  development  risks;  most  observers  suggest  that  it  is  the 
single  most  important  factor  in  the  engine  developments.  It  is  certainly  the  most  distinctive: 
Competition  rarely  continues  through  full-scale  development  into  production,  and  the 
competition  during  the  development  of  these  engines  was  unique.  The  Air  Force  and  its 
contractors  saw  competition  as  the  primary  element  of  the  developments  that  they  reacted  to 
directly.  They  also  saw  it  as  a  factor  that  shaped  m2uiy  specific  aspects  of  risk  management. 

Not  a  Formal  Competition 

When  we  think  of  competition  in  development,  we  normally  expect  a  competitive 
demonstrationyvalidation  stage  that  leads  to  a  prototype,  and  perhaps  a  formal  fly-off  to 


2For  good  accounts  of  these  efforts,  see  Drewes.  1987,  pp.  91-125;  Ogg,  1987,  pp.  58-74. 


-67- 


choose  one  contractor  for  full-scale  development  and  presumably  production.^  Such  a 
competition  provides  common  criteria  to  the  competitors,  similar  funding  to  complete  the 
demonstration  and  concept  validation,  and  similar  milestones  to  track  progress  and  make 
decisions. 

This  competition  included  none  of  these  features.  The  Air  Force  used  similar 
contractual  vehicles  to  fund  pre-full-scale  development  for  both  engines,  but  the  tasks, 
milestones,  and  especially  the  funds  used  were  quite  different.  The  Air  Fo*'ce  gave  GE 
substantial  pre-full-scale-development  iimds  just  to  get  up  to  speed  with  P&W  before  full- 
scale  development.  As  a  result,  the  Air  Force  spent  at  least  $150  railUon  or  65  percent,  more 
on  GE  than  on  P&W.  Treatment  was  more  nearly  similar  during  full-scale  development,  but 
even  here  we  see  major  differences.  While  GE  developed  one  engine  during  this  period,  P&W 
developed  three  engine  modifications  and  rolled  them  together  with  yet  another  modification 
to  develop  a  new  derivative  engine.  Specifications  and  criteria  for  the  two  developments 
were  quite  different.  The  draft  request  for  proposal,  the  first  formal  suggestion  of  what  the 
specifications  and  criteria  might  be  for  the  real  competition — the  production  competition — 
was  not  written  until  after  both  full-scale  developments  had  begun.  And  these  developments 
did  not  change  maukedly  as  the  production  competition  proceeded. 

The  competition  involved  during  development,  then,  was  not  consistent  with  any  form 
of  formal  competition  normally  prescribed.  It  was,  rather,  a  less  tangible,  but  no  less 
persuasive,  form.  As  far  back  as  1975,  observers  recognized  that  GE  might  successfully 
challenge  P&W  in  the  fighter-engine  market.  As  time  passed,  the  possibility  became  more 
serious.  The  Air  Force  and  its  contractors  recognized  the  high  stakes  involved  and  reacted 
accordingly.  Growing  anticipation  of  a  head  to-head  formal  production  competition  created 
an  intense  competitive  environment  throughout  the  developments,  even  though  the  Air  Force 
managed  no  formal  development  competition. 

Competition  Viewed  Broadly 

The  Air  Force,  GE,  and  P&W  each  saw  the  role  of  competition  lUfferently.  Whereas 
the  Air  Force  consistently  saw  it  as  a  way  to  reduce  development  risks  both  contractors  saw 
it  as  increasing  uncertainty  in  their  business  environments — GE  in  a  ;ositive  way  and  P&W 
in  a  negative  way.  To  understand  these  reactions,  let  us  consider  the  relationship  between 

'’Cuc-,  for  CoD  Directive  5000.1,  Policies  Governing  Defense  Acquisition,  15  August 

1990,  or  DoD  Instruction  5000.2,  Defense  Acquisition  Management  Policies  and  Procedures,  15  August 
1990. 
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the  Air  Force  and  P&W  before  GE  proposed  the  FIOIX  and  then  look  at  the  changes  GE’s 
entry  effected. 

For  most  of  the  1970s,  P&W  essentially  had  a  bilateral  relationship  with  the  Air  Force. 
Who  was  going  to  pay  to  fix  the  FlOO  was  an  issue  they  would  have  to  resolve  between 
themselves,  and  P&W  felt  confident  that  it  could  do  well  in  this  relationship.  The  FlOO  CIP 
allowed  P&W  to  pursue  a  set  of  solutions  with  Air  Force  funding  and  the  two  parties  were 
continuing  to  set  up  other  arrangements  to  fund  improvements.  For  P&W,  GE  appeared  as 
an  external  risk  to  this  relationship  in  the  late  1970s,  essentially  an  unexpected  alternative 
source  of  technological  capability.  P&W  managed  this  risk  by  trying  to  eliminate  it.  GE’s 
presence  seriously  threatened  the  effectiveness  of  P&W's  aggressive  stance  with  the  Air 
Force.  P&W  reacted  by  applying  this  stance  to  stamp  out  the  threat  and  preserve  their 
exclusive  relationship  with  the  Air  Force.  This  reaction  had  precisely  the  opposite  effect. 

GE’s  presence  sensitized  the  Air  Force  to  P&W’s  aggressive  attitude  and  gave  the  Air  Force  a 
potential  way  to  escape  it.  Acting  us  though  it  were  trapped  in  a  peiradigm  of  its  own 
making,  P&W  repeatedly  antagonized  the  Air  Force,  driving  it  toward  GE  and  ever 
increasing  the  risk  that  it  was  trying  to  eliminate. 

GE  saw  this  risk  from  the  other  side.  The  FlOlX  demonstrator  was  a  small-scale 
effort  with  a  very  large  potential  payoff.  It  offered  the  opportunity  to  make  a  large  inroad 
into  the  military  engine  market.  GE  nurtured  this  opportunity,  using  its  own  corporate 
funds  to  create  the  opening,  listening  carefully  to  what  the  Air  Force  said  it  wanted,  and 
funding  innovations  to  meet  Air  Force  needs  as  precisely  as  possible.*  The  approach  was  a 
stark  contrast  to  P&W’s.  In  fairness,  P&W  had  been  respon.sive  to  the  Air  Force’s  interest  in 
increased  thrust  in  the  FlOO  and,  after  the  Air  Force  made  additional  demands,  did  not  trust 
the  Air  Force’s  statements  about  what  it  wanted.  Nonetheless,  GE  listened  carefully,  with 
the  result,  under  these  circumstances,  that  it  succeeded. 

The  Air  Force  found  itself  as  being  between  the  competitors.  What  each  competitor 
saw  as  great  uncertainty  actually  reduced  the  Air  Force’s  perception  of  risk  in  the 
development.  Air  Force  managers  had  not  been  successful  in  getting  what  they  wanted  fi'om 
P&W.  They  saw  the  relationship  with  P&W  as  essentially  a  zero-sum  game,  in  which  P&W 
seemed  to  hold  the  cards  and  the  Air  Force  faced  the  prospect  of  very  high  costs  to  fix  the 
FlOO.  All  of  that  changed  when  GE  appeared.  The  Air  Force  saw  a  highly  motivated 
company,  willing  to  put  its  own  money  behind  its  ideas,  and  committed  to  making  the  FlOl 

••GE  was  seeking  a  way  to  enter  the  military  market  at  the  time.  It  wanted  a  way  to  increase 
the  scale  of  production  of  engines  that  it  could  then  produce  at  lower  unit  cost  to  sell  in  both  commercial 
and  military  markets.  For  more  information,  see  Drewes,  1987,  or  Kennedy,  1985. 
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DFE  succeed  to  open  the  door  to  the  fighter  market.  This  motivation  gave  the  Air  Force  a 
safe  contractor  to  work  with  in  its  new,  essentially  experimental,  EMDP.  And  it  gave  the  Air 
Force  a  stick  to  wave  at  P&W.  It  changed  the  balance  of  power  between  the  Air  Force  and 
P&W;  the  Air  Force  now  had  the  FlOl  DFE  to  force  P&W  to  make  concessions  that  the  Air 
Force  could  not  get  before.  And  if  P&W  did  not  come  through,  GE  looked  like  a  motivated 
alternative  that  could  deliver.  For  the  Air  Force,  then,  GE’s  appearance  changed  a  fairly 
high-risk  confrontation  with  P&W  into  a  lower  risk  opportunity  to  solve  serious  problems, 
one  way  or  another.  That  was  and  remains  the  primary  appeal  of  the  competition  to  the  Air 
Force. 

Relational  Contracts 

Seen  in  this  light,  the  competition  was  different  from  that  typically  encountered  in 
economics  textbooks.  The  primary  concern  of  the  buyer  was  not  to  use  competition  to  push 
product  price  down  to  margined  cost,  oJthough  this  was  certainly  a  consideration.  It  was  to 
change  the  balance  of  power  in  the  long-term  relationship  between  P&W  and  the  Air  Force. 
P&W  had  a  technological  and  production  capability  that  the  Air  Force  expected  to  exploit 
over  the  long  term.  (The  same  could  be  said  of  GE.)  Regardless  of  the  specific  terms  of  the 
contracts  the  Air  Force  would  sign  with  P&W  over  the  years,  the  Air  Force  expected  to 
maintain  the  relationship  and  use  it  to  mutual  advantage.  This  was  true  despite  the  extreme 
anger  that  many  Air  Force  officials  felt  toward  P&W  about  its  handling  of  problems  with  the 
FIOO. 

Such  a  long-term  relationship  is  governed  by  a  “relational  contract,"  an  implicit  set  of 
rules  that  allow  the  relationship  to  suivive  and  function  well  over  the  long  term.^  Such  rules 
Erne  rarely  formalized.  They  evolve  naturally  over  time  to  establish  a  sense  of  equity  in  the 
relationship  that  leads  both  parties  to  remain  in  the  relationship  voluntarily.  That  does  not 
mean  that  both  parties  benefit  equally;  they  can  still  benefit  in  proportion  to  the  power  they 
exercise  in  the  relationship.  The  developments  we  examine  here  offer  ample  evidence  that 
competition  can  significantly  alter  the  balance  of  power  in  such  a  relationship.  GE’s  entry 
shifted  power  towEU'd  the  Air  Force.  It  made  P&W  more  responsive  to  Air  Force  needs.  It 
also  reduced  the  Air  Force’s  need  to  monitor  P&Ws  actions  to  make  sure  they  were 
consistent  with  Air  Force  needs.  (P&W’s  presence  in  the  market  over  the  long  term,  of 
course,  also  assisted  the  Air  Force  in  its  continuing  relationship  with  GE.) 

5See,  for  example,  Macaulay,  1963;  Macneil,  1980;  and  Williainson,  1979. 
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Competltion  and  Responsiveness 

Air  Force  and  contractor  personnel  both  agree  that  the  most  important  efTect  of  the 
Alternate  Fighter  Engine  competition  was  to  increase  contractor  responsiveness  to  Air  Force 
needs.  Greater  responsiveness  led  the  Air  Force  to  perceive  lower  risk  in  the  program  as  a 
whole.  When  unexpected  events  occurred,  the  Air  Force  could  expect  greater  success  in 
getting  an  attractive  settlement,  thereby  reducing  the  probability  and  size  of  large,  negative 
effects.  Such  responsiveness  in  eflfect  shifted  risk  to  the  contractors  without  compensating 
them  for  it.  That  is,  one  consequence  of  competition  was  to  induce  contractors  to  bear  risk  on 
a  continuing  basis  at  a  lower  cost  to  the  government. 

The  increase  in  contractor  responsiveness  was  especially  evident  over  the  course  of  the 
production  contracts,  following  the  initial  source  selection  that  demonstrated  the  Air  Force’s 
seriousness  of  purpose  in  maintaining  competition.  The  production  contracts  were  set  up  to 
allow  a  series  of  annual  competitions  in  which  the  contractors  could  improve  the  terms  of 
their  offers  on  a  wide  variety  of  factors.  These  competitions  brought  small  concessions  on 
engine  unit  price,  the  factor  to  which  traditional  economic  arguments  give  greatest  attention. 
More  important  were  many  small  reactions  to  modernization  programs  and  modification 
programs  that  characterize  the  continual  negotiation  that  occurs  ur.der  a  relational  contract. 
Observers  agree  that  GE  and  P&W  were  both  much  more  responsive  in  these  negotiations 
than  they  were  in  similar  programs  in  which  they  felt  no  direct  competition. 

During  the  engine  developments  themselves,  competition  was  not  as  tangible  or  formal 
as  during  production.  But  even  here,  it  generated  similar  effects.  A  good  example  is  the 
continuing  set  of  contract-change  proposals  that  occur  in  any  development  as  information 
accumulates  and  the  parties  to  the  contract  adjust  their  mutual  goals.  The  Defense 
Acquisition  Regulations  set  out  formal  rules  for  making  equitable  adjustments  in  contracts  to 
reflect  contractual  changes.  But  those  rules  leave  considerable  room  for  negotiation. 
Competition  shifted  the  balance  of  power  toward  the  government  in  such  negotiations, 
allowing  the  Air  Force  to  get  many  contractual  changes  at  no  cost  to  itself  or  to  trade  one 
change  for  another  on  more  favorable  terms.  This  was  true  in  the  Air  Force’s  relationships 
with  both  GE  and  P&W.  But  it  was  probably  more  visible  in  the  relationship  with  P&W 
because  the  Air  Force  had  a  pre-competition  history  with  P&W  to  contrast  with  P&W’s 
behavior  under  competition.  In  fact,  following  its  stunning  success  in  the  early  production 
competitions,  GE  became  less  responsive  in  the  eyes  of  Air  Force  officials  charged  with 
overseeing  the  contract,  emphasizing  the  fact  that  competition  would  be  iraporteuit  in  the  Air 
Force’s  continuing  relationsliips  with  both  contractors. 
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A  second  specific  place  that  SPO  officials  believe  competition  improved  responsiveness 
was  in  the  FlOO  CIP,  in  which  important  parts  of  the  FlOO-PW-220  development  took  place. 
For  the  most  part,  the  FlOO  CIP  occurred  under  a  cost-plus-award-fee  contract.  Aw^l^d-fee 
contracts  allowed  the  Air  Force  to  determine  twice  a  year,  under  very  broad  guidelines,  what 
fee  the  contractor  should  earn  above  and  beyond  measured  costs.  The  parties  to  the  CIP 
preferred  such  an  arrangement  precisely  because  the  CIP  required  an  open,  flexible 
relationship  that  was  difficult  to  codify  in  detail.  In  the  past,  the  Air  Force  and  its 
contractors  could  not  identify  a  satisfactory  set  of  explicit  criteria  that  they  could  use  as  the 
basis  for  a  more  precise  incentive  arrangement.  The  Air  Force  considered  responsiveness  in 
determining  the  award  fee  and  believed  that  GE  and  P&W  actively  competed  in  their 
responsiveness  (and  other  behaviors)  to  best  one  another  in  the  size  of  this  award  each 
period.  That  is,  the  Air  Force  saw  more  responsiveness  under  this  contractual  arrangement 
when  competition  was  present  than  when  it  was  not. 

Such  increased  responsiveness  supported  the  flexible  decision  making  that  contributed 
to  the  success  of  these  developments;  we  address  this  issue  in  more  detail  below.  But  we 
should  not  overstate  the  effects  of  competition.  One  of  the  primary  goals  of  these 
developments  was  to  reduce  the  cost  of  ownership  for  engines  by  increasing  their  durability 
and  improving  maintenance  policies.  Air  Force  officials  believe  that  the  contractors  had 
strong  incentives  to  resist  any  effort  to  improve  reliability  or  durability,  incentives  that 
competition  alone  could  not  overcome.  Contractors,  as  the  primary  source  of  spare  parts  for 
engines,  had  an  incentive  to  design  engmes  that  would  not  seriously  threaten  a  continuing 
demand  for  spare  parts.  A  source  of  Air  Force  ill  will  toward  P&W  was  what  the  Air  Force 
perceived  to  be  a  P&W  policy  to  overprice  spare  parts  for  the  FlOO.  Given  that  policy,  the  Air 
Force  was  committed  to  changing  its  procurement  of  spare  parts  for  the  derivative  engines 
discussed  here.  Competition  did  encourage  GE  to  provide  data  packages  that  would  allow 
second  sourcing  to  reduce  monopoly-based  gains  from  the  spare  parts  market;  P&W  strongly 
resisted  such  arrangements.  Both  contractors  resisted  intensive  application  of  ENSIP 
methods  to  promote  Air  Force  cost-cutting  goals.  If  the  Air  Force  had  not  been  conversant  in 
ENSIP  methods,  competition  alone  probably  would  not  have  ensured  their  full  application. 

Competition  and  Opportunistic  Behavior 

One  important  characteristic  of  relational  contracts  is  that,  over  time,  the  participants 
make  investments — in  physical  capital,  skills,  techniques,  and  other  knowledge — that  are 
specific  to  the  rela\;ionship.  These  investments  are  less  valuable  outside  the  relationship 
than  in  it.  To  the  extent  that  these  assets  are  unique  to  the  relationship,  both  parties  have 
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an  incentive  to  do  things  that  allow  them  to  capture  the  full  value  of  this  uniqueness:  The 
parties  put  themselves  at  one  another's  mercy.  They  are  willing  to  do  so  because  the 
relationship  creates  enough  value  to  offset  the  resources  consumed  by  the  parties  in  their 
efforts  to  exploit  one  another.  But  such  efforts,  known  as  “opportunistic  behaviors,"  can 
potentially  consume  a  significant  portion  of  any  value  added  by  the  relationship.® 

A  development  program  is  a  particular  example  of  such  a  situation  in  which  the  two 
parties  agree  in  the  beginning  to  invest  resources  in  an  increasingly  unique  asset — the 
design  of  a  new  system.  As  time  passes,  more  and  more  resources  are  sunk,  pres  imably 
creating  an  asset  with  ever-increasing  value  over  time.  It  becomes  harder  and  harder  for 
either  party  to  withdraw  if  something  goes  wrong  and  more  and  more  tempting  to  seek 
changes  that  modify  the  terms  of  the  initial  agreement  on  the  share  each  party  has  in  the 
value  of  the  accumulating  asset.  In  the  absence  of  clearly  defined  controls  and  rules  that 
both  parties  understand  in  the  same  way,  risk  can  actually  grow  over  the  course  of  such  a 
development,  because  it  becomes  harder  and  harder  for  either  party  to  withdraw. 

The  Alternate  Fighter  Engine  competition  can  be  viewed  in  these  terms.  The  Air 
Force  became  increasingly  convinced  that,  despite  its  investment  in  P&W,  it  might  be  worth 
withdrawing  from  the  relational  contract  on  the  FIDO  in  response  to  P&W's  opportunistic 
behavior  within  that  contract.  P&W,  at  least  for  some  period  of  time,  did  not  consider  the 
threat  a  real  one  and  continued  its  opportunistic  behavior.  As  the  threat  became  more 
realistic,  P&W  saw  the  gains  it  had  achieved  through  opportunistic  behavior  being 
threatened  and  attacked  the  threat  rather  than  stopping  the  behavior  that  had  precipitated 
it.  That  is,  the  threat  of  competition  did  not  stop  P&W’s  opportunistic  behavior.  The 
realization  of  that  threat  in  the  source  selection,  however,  changed  everything. 

P&W  changed  not  only  its  behavior,  but  its  management.  The  result  was  a  basic  shift 
in  corporate  culture.  Part  of  this  was  reflected  in  greater  responsiveness,  as  suggested 
above.  Part  of  that  shift  was  reflected  in  fewer  efforts  to  exploit  its  relationship  with  the  Air 
Force.  In  fact,  the  presence  of  competition  fundamentally  changed  the  ability  of  P&W  or  GE 
to  engage  in  opportunistic  behavior.  The  Air  Force  had  made  very  lau-ge  investments  in  a 
large  fighter  engine,  but  it  could  procure  it  from  either  of  two  sources,  making  it  much  easier 
for  it  to  withdraw,  at  least  incrementally,  from  its  relationship  with  either  contractor.  If 
either  contractor  attempted  to  exploit  the  relationship,  the  Air  Force  could  turn  to  the  other 
contractor  without  reducing  by  much  the  value  of  its  owm  investments  in  the  large  fighter 
engine.  Neither  of  its  contractors  had  similar  alternatives.  The  net  result  was  that  the 


®For  an  extensive  discussion  of  such  behavior,  see  Williamson,  1985. 
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potential  for  opportunistic  behavior  by  either  contractor  dropped  as  competition  became  more 
tangible. 

The  Air  Force  responded  by  reducing  the  amount  of  monitoring  and  oversight  that  it 
had  found  to  be  necessary  in  the  past.  Arty  relational  contract  requires  knowledgeable 
participants  on  each  side  to  monitor  the  other  party  and  discourage  the  kind  of  exploitative 
behavior  suggested  above.  Competitton  does  not  eliminate  the  need  for  such  knowledgeable 
monitoring.  But  by  reducing  the  opportunities  for  exploitation,  it  reduces  the  need  for 
monitoring  and  closely  controlled  relationships.  In  particular,  it  allows  additional  flexibility 
in  the  relationship  to  respond  to  unexpected  events.  When  competition  disciplines  the 
plausible  responses  that  parties  can  have  to  a  surprise,  both  parties  can  proceed  with  less 
deliberate  caution  and  control.  The  net  result  is  that  they  can  move  ahead  together  more 
effectively  than  if  caution  is  paramount,^  Seen  in  this  way,  competition  can  play  r  profound 
role  in  risk  management.  Observers  of  the  developments  discussed  here  agree  that  it  played 
such  a  role.  In  this  role,  it  significantly  supported  program  managers'  abilities  to  exploit 
flexibility  to  promote  their  developments. 

Ironically,  competition’s  ability  to  discourage  opportunistic  behavior  also  limited  the 
Air  Force’s  ability  to  use  competition  to  do  so.  Although  not  well  understood  at  the  beginning 
of  the  pt  eduction  competition,  it  became  clear  that  the  Air  Force  valued  competition  enough, 
for  the  reasons  suggested  above,  that  it  was  not  willing  to  withdraw  entirely  from  its 
relationship  with  either  GE  or  P&W.  That  is,  both  competitors  came  to  appreciate  that  they 
could  engage  in  certain  opportu  oistic  behaviors  without  inducing  the  Air  Force  to  withdraw 
entirely.  Perhaps  the  clearest  indication  of  this  fact  was  the  Air  Force’s  response  to  P&W’s 
offer  in  the  first  production  competition.  P&W  priced  its  offer  in  a  way  that  strongly 
encouraged  the  Air  Force  to  buy  all  of  its  engines  from  P&W;  any  decision  to  buy  engines 
from  GE  significantly  increased  the  price  of  P&W’s  engfines  and  especially  of  its  warranty  on 
those  engines.  P&W  essentially  stated  in  that  offer  that  it  was  vrilling  to  bear  more  risk  with 
regard  to  its  warranty  if  the  Air  Force  dealt  exclusively  with  P&W,  effectively  reducing  the 
risks  to  P&W  associated  with  competition.®  The  Air  Force  accepted  a  large  increase  in  the 
price  it  paid  P&W  for  the  engines  it  did  buy  to  ensure  continuing  competition  and  thereby  to 
keep  both  contractors  at  risk.  Knowing  that  the  Air  Force  would  not  wholly  withdraw  from 

'The  argument  is  analogous  to  Schumpeter’s  discussion  of  an  economy  with  effective  controls. 
Controls  that  limit  responses  to  unexpected  shocks  in  the  economy  allow  the  economy  to  move  faster  in 
the  same  way  that  good  brakes  in  a  car  allow  the  car  to  proceed  at  higher  speed.  In  both  cases,  the 
“driver”  need  not  be  so  careful.  By  limiting  responses  to  surprises,  competition  enables  the  driver  in  a 
development  to  proceed  more  rapidly.  Schumpeter,  1950,  p.  84. 

®Cf  Drew'es,  1987,  p.  127. 
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®ithor  contractoi*  maintained  competition,  but  limited  its  positive  effects  (for  the  Air  Force) 
over  tho  developments’  duration.'* 

Summary 

Tho  Air  Force  did  not  develop  the  FlOO-PW-220  and  FllO-GE-100  engines  in  a 
forrauily  competitive  setting,  but  no  one  discoun*>id  the  presence  of  competition,  which 
became  more  tangible  nnd  influential  ns  the  developments  proceeded.  These  engines  are  best 
known  in  the  acquisition  community  for  the  role  that  competition  played  in  their 
development  and  production.  No  one  can  offer  quantitative  or  detailed  evidence  on  the 
effects  of  this  competition,  suggesting  that  carefiil  analysis  of  the  competition  itself  might 
be  fruitful.  We  can,  however,  offer  a  set  of  general  statements  about  the  competition  and  its 
effect  on  risk  nianagement. 

Although  GE  and  P&W's  growing  awareness  of  the  presence  and  importaime  of 
competition  appears  to  have  tempered  their  pricing  behavior  over  time,  its  meyor  effects  were 
quite  liilfereiit  from  tiiose  typically  emphasized  in  discussions  of  competition  policy.  While 
reducing  the  Air  Force's  general  risk  of  managing  serious  operability  and  cost  problems  in 
the  FlOO  fleet,  it  significantly  increased  FiStWs  risk  and  opened  a  mayor  opportunity  for  GE 
The  exjH*ctntion  and  final  realization  of  competition  in  production  fundamentally  altered  the 
nature  ufthc  relational  contrarta  between  the  Air  Force  and  its  engine  contractors,  GE  and 
F&W,  effectively  shifting  power  toward  the  Air  Force.  P&W  fought  this  change  as  GE 
promoted  it  Reaiizaiion  of  competition  and  its  attendant  uncertainties  tor  both  of  them 
ultimately  led  both  contractors  to  become  more  responsive  and  reduced  their  gains  froni 
pur.HUing  op[H)rtuiiistic  beha\ior.  In  this  lower  risk  setting,  the  Air  Force  found  its 
monitoring  job  significantly  simplifuHi  and  its  opportunity  for  ficx'^'le  aud  creative 
manageni'inl  significantly  enhvinewi. 

GENERAL  MANAGEMENT 

The  Propulsion  SPt)  bus  a  reputation  us  u  wuli-mn  organization.  As  noted  in  Section 
;i,  dot  el 'ipmeiil  uri-iigiiu's  has  iu'ighU'iied  the  SPO’s  awareness  of  the  importance  of  risk 
uiid  risk  muiiugenu  nt  Mumigors  in  the  SPO  see  their  general  management 
mission  as  inlieri'iitly  u  form  of  risk  manugi'nienl.  Even  when  the  engines  we  study  wore 
niidiT  di'M'lopnu'Ut,  those  inanitgeis  iiuili  on  theirjoint  experience  to  develop  a  set  ol' 

hot  diH.  UKni<'n  TMiMtH  quvttUiiiiii  uluui  liii?  b(>i>iifiLg  ol  full  luici  open  coiji|Aititicn  U6  curronlly 
piiiiituitiij  tiy  ill*'  apinl  ul  till'  ( 'iim(>«lilioii  in  ( 'I'nlmvling  Act  of  lUrit.  The  llndiiig*  r*|><irt4id  here  ure 
oiiii|irtiilile  Ntiih  ilintke  III  Kehiidii,  IWUO,  .ihirii  raiiw  M>riiiu>i  doohli  sIhiuI  piirsuiiig  uiifulleo'd 
igiii|K'Uliiiii  Hni.iiiy  veMiioM)  lo  tile  I'  S  guvemiiieiit.  end  t iuiMi  in  .Siuiikc,  IlMn,  wtiicli  raise  mIiiiIIui 
ili.iihli.  Ill  a  priN  ati'M'i  lor  ■....-ileM  I  uii.  cuiKiiiuliig  to  examine  thin  question  in  ongoing  miulyxih. 
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standard  operating  arrangements  that  produced  substantial  structure  without  overwhelming 
developers  with  bureaucratic  requirements.  In  fact,  the  order  provided  by  the  managers’ 
experience  and  procedures  appears  to  have  promoted  flexibility  and  creativity.  This 
subsection  reviews  key  factors  in  the  Propulsion  SPO's  general  management  approach  during 
the  development  of  the  FlOO-PW-220  and  FllO-GE-100  and  presents  examples  of  flexibility 
and  creativity  promoted  by  this  approach. 

Experience 

When  the  Air  F  orce  formed  the  propulsion  SPO  in  1977,  it  consolidated  all  of  the 
engine  programs  in  the  Aeronautical  Systems  Division,  bringing  together  a  group  of 
managers  and  engineers  with  many  years  of  experience  overseeing  the  development  of 
engines.  Personnel  from  the  F-16  SPO  transferred  to  oversee  FlOO  programs,  including  the 
EMDP  that  began  in  1979,  The  chief  engineer  for  the  FlOO  program,  Mr.  James  Day,  had 
been  a  manager  in  the  FlOO  program  from  its  inception.  Personnel  who  had  worked  on  the 
FlOl  development  were  available  for  the  FlOl  DFE  program.  For  example,  Mr.  Joseph 
Wood,  the  program  manager  for  the  FlOl  DFE.  had  been  chief  engineer  for  the  FlOl.  His 
chief  engineer  for  the  FlOl  DFE,  Mr.  Joseph  Batka,  was  an  engineer  from  the  Propulsion 
Laboratories  who  had  already  spent  several  years  overseeing  the  analysis  of  risk-related 
issues  on  the  FIOIX,  including  inlet  compatibility  of  the  F-i6  and  FlOl  DFE,  software  that 
the  now  closed-cnd-loop  control  system  in  the  FlOl  DFE  would  use  to  eliminate  stall- 
btagnatio;.:.,  ,uid  statistical  models  relevant  to  ENSIP  and  the  estimation  of  operating  and 
.support  costs. 

Such  personnel  formed  the  backbone  of  the  development  efforts’  formative  stages 
They  formed  the  core  of  tlie  development  staff  over  the  developments’  diirnt’on.  Four  SPO 
commanders  played  a  role  in  these  developments  through  1986,  none  with  more  than  three 
years  in  place.  But  the  primarily  civilian  stuff  they  commanded  remained  in  place  as  thijy 
came  and  went.  The  civilian  managers  brought  with  them  clour  understanding  of  the 
reasons  the  SPO  hud  been  organized,  reasons  closely  related  to  the  goals  set  for  the  new 
ongines.  They  sought  fundamental  changes  in  the  Air  Force's  approach  to  engine  durability, 
operiibility.  and  .supportahility.  The  engines  we  .study  were  the  first  directly  uffected  by  this 
new  interest,  but  it  curried  well  beyond  these  developments  alone.  The  entire  SPO,  not  just 
the  sections  of  it  committed  to  the  two  engincM,  inainluined  an  interest  in  a  new  approach, 

Similar  continuity  existed  ill  the  coatruelors.  As  gnnerul  mnnugisr  of  the  FlOO  I'umily 
iil'eiigiiieH  at  I’&W,  Mr.  Hd  Ford  over.sow  the  FlUO-l’W-220  and  iU  iininediute  precursors 
Irom  U)7l)  on.  t  smtinnity  in  the  FlOO  prograni  leading  up  to  these  de , ,  lopiiiuiUs  allowed 
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P&W  to  maintain  its  key  staff.  At  GE,  Mr.  Brian  Brimelow  worked  on  the  FlOl  DFE  and 
then  manaeed  the  FlOO-GE-100  full-scale  development.  GE  maintained  key  personnel  from 
the  FlOl  development  following  cancellation  of  the  B-1  procurement  and  worked  with  the  Air 
Force  to  create  a  Transition  to  En^neering  Development  that  maintained  the  staff  during 
the  move  from  the  FlOl  DFE  EMDP  to  the  PllO-GE-100  full-scale  development. 

From  the  very  beginning,  the  Propulsion  SPO  employed  a  matrix  structure  that 
enhanced  the  accumulation  and  application  of  experience  during  the  developments.  Table 
5.1  shows  the  m^or  offices  in  the  matrix  at  two  points  before  and  after  1981,  when  the  SPO 
experienced  the  one  significant  reorganization  that  occurred  during  the  developments. 
Functional  offices  allowed  expertise  to  accumulate  for  application  to  all  programs.  Although 
such  accumulation  was  limited  before  the  developments  began  in  1979,  it  occurred  much 
more  rapidly  over  their  lives  than  would  have  occurred  if  these  were  the  only  developments 
that  the  SPO  handled  during  this  period.  The  SPO  handled  a  number  of  other  developments 
during  this  period,  and  those  rapidly  expanded  experience  in  the  engineering,  test,  contract, 
program  control,  and  other  functional  offices  key  to  development.  On  the  program  side  of  the 
matrix,  EMDPs  begun  in  the  new  programs  office  (YZN)  transited  smoothly,  with  relevant 
personnel,  to  the  main  program  offices  as  the  developments  progressed.  These  offices 
maintained  unity  in  the  programs  while  drawing  on  the  accumulating  functional  expertise 
whenever  necessary. 

Table  6.1 

Matrix  Structure  of  the  Propulsion  SPO 


1979 

1964  1 

Programa 

FlOO  joint  engine  project  office 

Tactical  engines  program  office 

F107  joint  engine  project  office 

Strategic  engines  program  office 

In-service  joint  engine  project  office 

Airlift  and  trainer  engines  program  office 

New  engine  project  office 

New  engines  program  office 

Function* 

Coiimiaiider's  office 

Commander's  office 

Management  operations 

Management  operstioni 

Program  control 

Program  control 

Configuralinn  management 

Acquisition  support 

Contracting  and  manufacturing 

Contracting 

Manufacluring/quullty  nssurance 

Propulsion  logistics 

Pi'ouulslon  logistics 

Engineering  and  lest 

Engineering 

SOUKCE.S  U.S.  Air  Korce,  Aviviuulicit  byntwiiu  DIvitIun  Dupuly  Inr  Pt'iipulNlon  (irsttiiizslltiii 
chatXii,  Oflulmr  IHVU,  OcUilwr  1HM4. 
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This  matrix  and  the  personnel  working  in  it  maintained  a  special  kind  of  experience 
that  was  important  to  our  two  developments— ctimulative  experience  with  the  developments’ 
contractors,  P&W  and  GE.  The  Air  Force  actually  organized  the  contracting  section  of  the 
SPO,  YZK,  into  sections  along  contractor  lines,  effectively  one  for  P&W  (YZKA)  and  one  for 
GE  (YZKB).  In  1981,  the  Air  Force  reorganized  the  program  side  of  the  SPO  matrix  along 
similar  lines,  with  a  “tacticar  engines  office  (YZF)  to  deal  with  P&W  and  a  ‘'strategic” 
engines  office  (YZY)  to  deal  with  GE.  The  engineering  office  was  also  split  along  similar  lines 
(YZEF  and  YZEY).  This  structure  allowed  the  SPO  to  bring  considerable  experience  with 
these  contractors  to  bear  across  programs  that  the  contractors  worked  on.  Discussions  with 
SPO  officials  reflect  an  intense  appreciation  of  differences  in  the  cultures  of  these  two 
contractors,  an  appreciation  that  transcends  and  informs  the  two  developments  we  examine. 

Looking  beyond  the  formal  matrix  for  the  SPO,  the  Air  Force  engine  development 
community  also  included  the  Arnold  Engineering  Development  Center  (AEDC)  and  the 
Flight  Test  Center  (AFFTC)  at  Edwards  Air  Force  Base.  The  SPO  maintained  long-term 
relationships  with  those  centers  and  had  well-estabKjhed  procedures  for  dealing  with  them. 
During  the  developments  we  study,  problems  occasionally  arose  between  the  SPO  and  these 
centers,  but  the  problems  were  e5:ceptional  and  were  solved  quickly.  Contractual  material 
and  interviews  suggest  that  such  experience  is  typical;  the  SI  0  had  a  smooth-running 
relationship  with  the  principal  government  facilities  it  used  for  testing.  The  Navy  and  NASA 
also  conducted  tests  during  our  developments.  Although  those  tests,  too,  appear  to  have  gone 
smoothly,  the  SPO  did  not  typically  work  closely  with  these  centers. 

The  final  link  in  the  Propulsion  SPO’s  experience  base  was  the  rapidly  developing 
formal  development  doctrine  that  it  used  to  support  ito  general  risk-management  goals  and 
its  goals  in  the  two  developments.  The  SPO  was  already  using  accelerated  mission  tests  in 
the  late  1970s  to  bring  empirical  evidence  to  bear  on  the  durability  issue.  It  was  using  the 
four-step  process  to  extend  the  development  cycle  to  further  emphasize  durability, 
operability,  and  supportability  and  to  reduce  concurrency  between  development  and 
production.  And  it  was  formalizing  a  set  of  technical  insights  into  the  engineering  structural 
integrity  program  (ENSIP),  a  set  of  tools  that  found  their  first  application  in  the  FIOO-PW- 
220  and  FllO-GE-lOO  developments. 'These  analytic  devices  provided  a  formal  approach  to 
risk  management  that  helped  structure  the  derivative  developments  we  study. 
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Standard  Procedures 

As  a  standing  SPO  that  remains  in  place  as  developments  come  and  go,  the  Propulsion 
SPO  has  developed  many  standard  operating  procedures.  A  number  of  them  appear  to  have 
been  especially  important  during  the  developments  we  study:  program  business/technical 
review,  independent  propulsion  review  teams,  the  test-analyze-fix  approach  to  development, 
contracting  procedures,  and  generad  procedures  derived  from  the  CIP  approach  to 
development. 

The  program  business/technical  review  (PB/TR)  every  other  month  reviewed  progress 
against  key  contractual,  control,  engineering,  and  test  milestones  for  each  development.  By 
surfacing  potential  problems  and  considering  potential  solutions  that  cut  across  functional 
areas,  it  facilitated  the  SPO’s  execution  of  integrated  system  development  in  a  matrix 
setting.  A  standard  management  device  unlikely  to  be  unique  to  the  Propulsion  SPO,  the 
PB/TR  was  designed  to  pass  a  thorough  overview  of  each  development  above  the  program 
office  on  a  regular  basis.  It  also  played  the  critical  role  in  a  matrix  organization  of  ensuring 
that  all  parts  of  the  organization  were  current  on  the  status  and  needs  of  each  development 
in  progress.  Managers  of  each  three-digit  office  (the  offices  shown  in  Table  5.1)  attended. 
Hence,  the  reviews  ensured  that  all  functional  offices  saw  the  development  programs  in  a 
similar  light  and  understood  constraints  imposed  by  each  of  the  functional  capabilities;  that 
different  program  offices  involved  with  the  developments  maintained  a  common  image  oi  the 
developments,  facilitating  when  appropriate  hand-off  of  tasks  and  personnel  from  one  office 
to  another;  and  that  related  programs,  such  as  the  FlOl  and  FllO  programs,  had  some 
formal  interaction  even  if  they  were  not  coordinated  on  a  day-to-day  basis.  They  also  added 
cumulative  knowledge  on  the  SPO's  experience  with  each  of  the  key  contractors. 

To  complement  the  regular  internal  review  of  issues  in  the  program  business/technical 
review,  the  Propulsion  SPO  used  ad  hoc  independent  propulsion  review  teams  or  gi  oups  to 
examine  material  developed  by  the  SPO  and  its  contractors  before  major  decisions  wore 
made.  Such  u  team  included  u  highly  respected  chairman  and  membors  from  outside  the 
primary  development  team.  The  SPO  regularly  used  such  teams  before  committing  to  and 
starting  a  flight-test  program.  It  also  used  them  when  major  problems  arose  in  a 
development  and  the  SPO  did  not  feel  confident  resolving  them  on  its  own.  During  the 
developments,  such  team.s  rarely  found  significant  problems  in  SPO  plans.  They  did  raise 
lesser  issues  that  the  SPO  and  its  contractors  could  then  manage  before  an  accident  resulted. 
And  they  provided  political  cover  for  the  SpO  commander  in  case  anything  Herious  did  occur. 
As  luck  would  have  it,  tho  SPO  never  requirod  such  cover  during  the  two  engine 
develo|>mentH. 
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The  development  process  itself  wa?  seen  as  highly  empirical  and  pragmatic.  The  test- 
analyze-fix  (TAF)  approach  was  well  accepted  and  established  when  the  developments  began, 
and  the  SPO  maintained  it  throughout  the  development.  “  Formal  weekly  activity  reports, 
command  assessment  reviews,  and  SPO  office  reports  produced  during  the  developments 
reflect  this  cumulative  approach  by  chairacterizing  the  developments  in  terms  of  week  after 
week,  month  after  month  of  almost  tedious  test  sequences  to  locate  and  fix  problems.  This 
approach  takes  time:  Weapon  system  developers  typically  expect  an  engine  development  to 
take  longer — two  to  three  years  longer  for  a  centerline  engine — than  that  for  a  matched 
airframe  development,  in  part  because  of  this  data-intensive,  progressive  approach  to 
development.  By  its  very  nature,  however,  the  TAF  approach  tends  to  yield  robust  results. 

It  tests  and  fixes  an  engine  until  the  engine  stabilizes  in  a  design  that  the  Air  Force  can  use. 
As  long  as  the  TAF  approach  applies  tests  relevant  to  the  expected  usage  pattern  for  the 
engine,  the  approach  identifies  and  resolves  most  risks  relevant  to  design,  development,  and 
final  product  verification. 

We  should  not  conclude  from  this  description  of  a  rough-and-ready,  robust  test  regime 
that  it  was  crude  or  clumsy  in  any  way,  On  the  contrary,  each  development  routinely  ran 
series  of  tests  in  parallel,  often  at  several  different  test  facilities,  and  regularly  coordinated 
the  activities  in  response  to  unexpected  events. 

A  good  example  on  the  engine’s  implementation  is  a  test  failure  that  several 
participants  identify  as  a  m^or  event  in  the  FlOl  DFE  development.  During  the  fifth  test 
flight  for  the  FlOl  DFE  in  an  F-16  at  Edwards  Air  Force  Base,  California,  on  7  January 
1981,  the  pilot  and  a  chase  plane  detected  what  appeared  to  bo  a  fuel  leak.  Following  an 
emergency  landing,  inspecto’**'  found  that  a  90-deg  aluminum  elbow  fuel  tube  that  connected 
the  engine  to  the  aircraft  fuel-supply  system  had  cracked.  The  damaged  part  was  at  the  GE 
facility  in  Evendale,  Ohio,  the  following  day  to  allow  an  engineering  team  from  ASD  to 
inspect  it. 

Within  a  week,  GE  had  determined  that  the  tube  experienced  higher-than-anticipated 
loads,  duplicated  the  failure  on  a  fuel-system  rig,  and  verified  that  a  support  link  would 
reduce  those  loads.  Developers  decided  to  add  a  support  link  and  to  replace  the  aluminum 
tubo  with  a  stuinloss-steol  tubu  to  udd  strength.  Following  tests  of  the  proposod  fix  in  un 
engine  cell  at  Evendale,  the  FlOl-DFE-powered  F-16  resumed  flight  at  Edwards  (on  31 
January)  with  a  restriction  to  nonaugmented  flight.  Work  continuod  to  roduco  pressure 

"Sfclloii  .'I  dlscuHriOH  lluH  appniucli  In  Ihu  coiUuxl  of  the  Air  Furcu's  risk  munugeriiurit  doctrine. 

Air  Furco,  Acnuuiutlcul  .SyNlciim  Division,  "Doputy  for  I'ropul-  inn  Weekly  Activity 
Kopurl,"  H  January  IPHl. 
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spikes  from  afterburner  pump  initiation  by  releasing  fuel  more  slowly  to  fill  the  engine 
afterburner  pump  and  modifying  the  aircraft  fuel  inlet  pipeline.  This  second  change  required 
coordination  with  General  Dynamics,  the  airframe  manufacturer.  Ground  tests  on  the  new 
configuration  were  completed  at  Evendale  on  24  February.  Following  review,  of  the 
generated  data,  by  the  Propulsion  and  F-16  SPOs  and  ASD’s  Engineering  Office,  the  aircraft 
was  released  for  one  unrestricted  flight,  to  be  followed  by  further  analysis.  By  10  March,  the 
aircraft  had  returned  to  normal  flight-test  activities. Two  months  after  a  serious, 
unexpected  test  incident,  the  developers  had  coordinated  two  contractors,  two  SPOs,  two  test 
facilities,  and  additioned  ASD  personnel  to  design,  test,  and  implement  a  robust  solution  that 
could  be  used  in  the  final  engine  configimation  if  necessary  and  that  kept  the  test  program  on 
schedule. 

In  a  similar  manner,  the  Propulsion  SPO  was  successful  in  coordinating  development 
activities  across  programs  within  the  SPO  and  even  across  the  services.  Informal  interaction 
and  devices  such  as  the  program  business/technicaJ  review  handled  routine  coordination. 

The  SPO  also  stood  ready  to  manage  less  predictable  and  often  serious  problems. 

For  example,  the  failure  of  an  F404  turbine  disk  made  of  Ren6  95  caused  the  crash  of  a 
Nav\'  F-18  in  September  1980.  The  FlOl  DFE  had  several  parts,  including  disks  and  shafts, 
made  of  Ren6  95.  Before  flight-testing  the  FlOl  DFE  in  an  F-16,  the  SPO  used  the  available 
F404  data  to  conduct  an  ENSIP-based  risk  assessment  to  determine  the  expected  time  to 
failure  in  the  relevant  components  in  its  test  engines.  It  concluded  that  “all  FlOl  DFE  R’95 
part  life  predictions  exceed  the  planned  flight  test  usage,”  and  subsequently  proceeded  with 
the  test.  New  Navy  data  on  the  F404  surfaced  on  11  May  1981,  prompting  the  SPO  to  stand 
down  F-16/F101  DFE  flight  operations  while  it  reassessed  the  safe  operating  life  for  the  FlOl 
DFE.  The  SPO  completed  the  analysis  within  four  days,  allowing  flight  tests  to  resume  on  16 
May.'^  No  formal  channels  existed  to  coordinate  and  act  on  such  information  with  the  Navy; 
the  Propulsion  SPO  handled  che  situation  calmly  and  efficiently. 

Such  coordination  was  routine  on  FlOO  development  programs,  which  regularly 
exchanged  engines  in  a  way  that  would  only  be  possible  with  close  coordination  across 
individual  development  efforts.  Coordination  did  not  work  as  well  on  the  FI  10  side  of  the 
hous(!.  The  FlOl  DFE  program  set  up  a  common-issues  program  with  the  FlOl  program,  but 
interaction  v^as  iimitod.  The  SPO  sot  up  a  liaison  function  betwcon  the  FlOl  and  FllO 

'■'U.S.  /\ir  I'urco,  Auronaullcul  .SyMtimiM  Dlvimoii  “Deputy  for  l^ropulsluii  Wcokly  Activity 
lioport,"  l/i  110110017  1981,  n  I'Vhruury  19HI,  12  I’ubruiiry  1981,  19  Kebruury  1981,  2(1  Kebruury  1981,  12 
Murcli  1981, 

HU.S.  Air  Korto,  Auronoutlciil  .Sy^U'uui  Divluiou,  'Deputy  for  I’ropulHion  Wtiokly  Anivily 
luipmt,"  2(1  I'uliruury  1981,  l-l  May  1981,  21  Muy  1981. 
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programs  during  the  early  years  of  FI  10  production;  the  liaison  function  closed  down  shortly 
thereafter.  When  the  B-IB  program  restarted,  it  did  so  under  such  tight  schedule  and  cost 
constraints  that  the  Propulsion  SPO  elected  to  isolate  the  FlOl-GE-102  engine  being 
developed  Tor  the  B-IB  from  related  programs  to  simplify  the  engine’s  management.  This 
action  may  have  simplified  the  management  of  the  closely  related  FlOl  DFE  and  FllO-GE- 
100  as  well,  but  it  also  sacrificed  significant  opportunities  for  coordinating  what  could  have 
been  complementary  test  programs.  Coordination  might  have  increased  the  external  risk 
that  each  development  faced,  but  should  ultimately  have  reduced  the  joint  risk  that  the  two 
programs  faced  alter  spending  any  given  level  of  time  and  resources;  that  is,  the  decision  not 
to  coordinate,  for  better  or  worse,  ultimately  increased  the  risk  to  the  SPO  as  a  whole. 

Similarly,  GE  developed  the  F118  engine  for  the  B-2  from  a  core  common  to  the  FlOl 
DFE  and  FllO.  Because  F118  development  proceeded  in  a  “black”  program,  little 
coordination  was  possible,  in  the  SPO  or  at  GE.  More  generally,  coordination  at  GE  among 
military’  engine  programs  worked  well  in  the  absence  of  explicit  restrictions.  Coordination 
between  the  military  FlOl,  FllO,  and  F118  programs  and  the  technologically  related  civilian 
CFM-fi6  program  remained  weak  despite  some  SPO  effort  to  enhance  it.  This  problem 
apparently  persisted  when  the  Air  Force  began  to  buy  CFM-66  engines  itself  for  the  C-135. 

Even  within  a  development  program,  significant  coordination  was  required  as 
development  passed  from  one  contract  to  another.  Such  passages  were  most  common  in  the 
CIP,  for  which  new  contracts  began  each  year  in  the  early  part  of  the  developments  we 
examine.  The  contracts  smoothly  transferred  development  tasks  and  test  assets  from  one 
contract  to  the  next.  The  absence  of  significant  modifications  and  the  smooth  administration 
of  the  award-fee  process  point  to  a  process  that  functioned  well  during  the  period.  SimilEU" 
transfers  from  EMDP  to  frill-scale  development  contracts  point  to  a  similarly  smooth  process 
being  in  place. 

The  SPO  staff  generally  believes  that  the  Defense  Acquisition  Regulations  (DAR),  the 
contract  law  that  it  uses  to  administer  its  relationship  with  its  contractoro,  were  (and  are) 
more  than  adequate  for  the  task.  One  manager  noted  that  continual  reforms  in  that  law 
typically  creiile  more  turbulence  and  risk  than  they  resolve  by  muddying  the  precedents  and 
the  informal  relational  contract  that  the  SPO  uses  to  manage  its  relationship  with  its 
contractors.  During  the  devclopment.s  we  examine,  such  turbulence  appears  to  have 

did  nut  dulurniint)  tlic  reason  fur  this  woukiiess.  It  dusurves  further  attention, 
slinilur  cutnpluiiit  is  heard  with  regard  to  chungu  in  civil  law  in  general.  Many  of  ihu 
arguments  olfored  about  civil  law  apply  to  the  DAK  without  diiTicully. 
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created  only  one  small  problem,  a  problem  that  the  SPO  and  P&W  satisfactorily  resolved 
without  serious  difficulty. 

The  CIP,  which  predated  the  Propulsion  SPO  by  many  years,  created  a  set  of  stauidard 
procedures  that  were  well  developed  and  functioning  smoothly  by  the  time  the  subject 
developments  began.  At  the  center  of  these  procedures  was  (and  is)  the  Resolution  of 
Operational  Problems  (ROP)  clause,  which  stated  that  “it  is  recognized  by  the  parties  hereto 
that  the.  .  .Component  Improvement  Program  is  dynamic  and — therefore — it  is  anticipated 
that  tasks  will  be  added  auid  cancelled  during  the  course  of  the  contract.’’*’^  The  CIP  contract 
identifies  clear  priorities  and  procedures  for  adding  and  removing  tasks  as  development 
proceeds  or  as  external  events  change  funding  or  priorities  within  the  CIP.  Using  these 
procedures,  the  Air  Force  can  expedite  the  administration  of  changes  in  a  development 
program.  The  net  result  is  an  orderly  process  that  reduces  many  of  the  risks  often  associated 
unth  flexible  program  administration  and  hence  encourages  flexibility.  The  process  is  not 
risk-free.  Both  the  Air  Force  and  its  contractors  have  yielded  to  the  temptation  to  exploit 
this  flexibility  in  inappropriate  ways.  But  over  the  long  run,  the  program  has  been  quite 
successful  at  allowing  timely  resolution  of  development  issues. 

The  CIP  played  an  important  but  limited  direct  role  in  the  developments  we  study. 

But  it  provides  a  tradition  for  the  Propulsion  SPO,  with  eftects  that  extend  well  beyond  the 
CIP  itself.  All  SPO  personnel  are  familiar  with  the  CIP  and  value  the  flexibility  that  it 
allows.  It  provides  a  model  for  administering  development  that  CEUinot  be  transferred 
wholesale  to  the  rest  of  the  SPO,  but  does  color  the  way  the  SPO  approaches  development. 

In  the  end,  the  CIP  also  serves  as  a  useful  metaphor  for  the  Propulsion  SPO's  general 
approach  to  managing  development.  Both  the  CIP  and  the  Propulsion  SPO  have 
accumulated  experience  in  managing  engine  development  over  numerous  engine  designs. 
Both  use  well-conceived  standard  procedures  for  monitoring  and  reacting  to  unexpected 
events.  As  a  result,  managers  in  both  the  CIP  and  the  Propulsion  SPO  have  the  confidence 
and  tools  req\iired  to  pursue  a  fairly  flexible  and  creative  approach  to  development.  The  next 
subsection  considers  iomu  examples  that  illustrate  this  point. 

Illustrctlons  of  Flexibility  and  Creativity 

The  FlOO  EMDP  was  initially  set  up  to  introduce  demonstrated  advanced  component 
technologies  into  the  FlOO  production  engine  design.  One  of  the  two  koy  components  initially 

'■'U  S.  v\ii'  Furcu,  Auroimutlcul  Syaluiiis  Division,  Contract  F33667-81-C-0001,  Wriahl-FatUirsoii 
Air  Forcu  liusu,  Ohio,  14  OcCobor  l‘JHU,  Section  li,  Cluusu  1(1,  p.  '22.  .Slinllur  lunguago  oxlsts  in  all  CIP 
cunlrucls. 
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considered  was  a  partial  swirl  augmentor.  V/ithin  four  months  of  the  FlOO  EMDP’s 
initiation,  altitude  tests  at  the  NASA  Lewis  ground  engine  test  facility  had  failed  to 
substantiate  any  of  the  operability  benefits  claimed  for  this  “previously  demonstrated"  and 
hence  presumably  low-risk  component.  For  lack  of  funds,  the  Air  Force  could  not  begin  its 
own  tests  at  A£DC  until  March  1980,  and  foreign-object  damage  to  a  test  engine  delayed  the 
tests  still  further.  The  Air  Force  completed  a  preliminary  design  review  of  the  partial  swirl 
augmentor  by  June  1980,  but  later  Air  Force  tests  produced  results  similar  to  the  NASA 
results.  By  the  end  of  1980,  P&W  and  the  SPO  had  reacted  by  replacing  the  partial  svnrl 
augmentor  with  another  advanced  design,  a  ducted  flameholder  configuration.  Despite  these 
difTiculties,  the  failure  of  the  partial  swirl  augmentor  ultimately  had  no  negative  effects  on 
the  program  because  an  alternative  was  readily  available  to  replace  even  this  fairly  low-risk 
item.‘® 

The  beginning  of  the  FlOl  DFE  EMDP  demonstrates  a  remarkable  degree  of 
creativity.  With  no  funding  of  its  own  to  commit,  the  Air  Force  located  unused  funds  that 
Congress  had  appropriated  for  the  Navy.  Despite  considerable  difficulties  following  the 
Na\7's  untimely  earlier  withdrawal  from  the  FlOO  development,  a  withdrawal  that  required 
major  readjustments  in  that  program,  the  two  services  were  able  to  reach  agreement  over  an 
arrangement  that  would  transfer  $33  million  of  $41  million  to  the  Air  Force  and  give  the  Air 
Force  effective  control  over  the  development  so  funded.  Funding  remained  tenuous 
throughout  this  EMDP,  but  this  first  step  initiated  a  process  that  ultimately  yielded  all  the 
funds  required  for  the  development  without  significantly  affecting  the  progress  of  the 
development  effort  itself  In  the  last  step  of  this  EMDP  process,  the  developers  established  a 
new  type  of  program,  a  Transition  to  Engineering  Development,  that  allowed  GE  engineers 
to  continue  the  development  process  beyond  the  initial  EMDP  plan  while  the  Air  Force  built 
support  for  a  full-scale  development.  This  effort  kept  the  GE  design  team  together  and 
allowed  a  smooth  transition  from  the  initial  demonstration  into  a  final  product-verification 
program. 

The  FlOl  DFE  EMDP  was  set  up  as  a  “success-oriented"  program,  meaning  that  it  had 
less  than  the  normal  amount  of  slack  allowed  in  a  development  program  to  deni  with 
unanticipated  events.  Us  managers  perceived  that  any  major  failure  in  the  program  could 
kill  it.  Thu  Air  Force  was  willing  to  use  this  approach  in  part  because  killing  the  program 

>M()ur  (loBcriplion  rcliuH  la  purl  on  ufficiul  Air  Force  hislorioH  of  various  Aeronautical  SysLurns 
Divinion  dovulopimml  prograniH  wrillon  by  ti.  .1.  (ioigur  during  tU79-lUHU.  l.'ueuusii  thoMo  uncluHwlliod 
liistorius  an!  chaplorh  in  clasMiliud  Air  Force  documunls,  we  cannot  proviue  explicit  clUilionH  in  this 
ducuinoni.  Liipubliubud  liiniory  reports  of  thu  I'rupulHiun  SPO  Now  Fugiiiua  Program  Officu,  Junuury- 
Junu  PJHO,  p.  11-5,  and  .luly-DccomlH'i  IDHO,  p.  H-b. 
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would  not  threaten  mcgor  Air  Force  goals  and  because  GE,  to  ensure  that  the  program  as  a 
whole  succeeded,  had  very  strong  incentives  to  back  up  the  program  in  case  of  test  failures. 
But  the  Air  Force  did  not  rely  on  GE  alone  to  ensure  success.  Tlie  SPO  used  unusually  close 
oversight  of  the  ongoing  test  program  and  continually  informed  its  superiors  about  the 
program's  successes  and  failures.  The  SPO  worked  hard  to  prevent  the  development  of 
rumors  that  could  magnify  routine  difficulties  out  of  proportion.  This  approach  is  reflected  in 
the  SPO  commander’s  weekly  activity  reports  to  the  ASD  commander.  These  reports  gave 
the  FlOl  DFE  EMDP  disproportionate  attention  and  often  provided  minutely  detailed 
coverage  of  problems  and  efforts  to  resolve  them  from  one  week  to  the  next.  These  reports 
also  make  clear  that  frequent  communication  of  import  ant  information  on  problems  was 
common. 

The  environment  for  the  FllO  full-scale  development  was  more  comfortable  and 
forgiving.  The  development  managers  were  able  to  build  important  slack  into  the  program 
that  allowed  successful  adaptation  to  a  number  of  surprises. 

For  example,  as  the  development  proceeded,  GE  fell  farther  and  farther  behind  in  its 
test  schedule,  completion  of  nontest  activities,  and  delivery  of  data  items  to  the  Aj.”  Force. 

GE  proposed  a  number  of  solutions  that  it  could  not  execute.  It  also  became  apparent,  as 
product-verification  activities  fell  behind  their  schedule,  that  the  schedule  for  such  activities 
at  AEDC  was  unrealistic.  The  SPO  sought  ways  to  intervene  at  GE  and  increased  the 
priority  of  FllO  test  activities  at  AEDC.  In  the  end,  the  SPO  could  not  avoid  slipping  the 
deadline  for  product  verification  from  30  September  1984  to  31  January  1985.  This  was  a 
significant  slip,  since  the  product-verification  milestone  was  mutually  set  at  the  start  of  full- 
scale  development  only  nine  months  earlier.  But  it  did  not  affect  GE's  scheduled  delivery  of 
engines  to  the  airframe  manufacturer.^® 

The  SPO  and  GE  absorbed  the  slip  by  using  a  shorter  interval  to  review  the  results  of 
the  product-verification  process  in  anticipation  of  production.  To  make  the  planned 
production  schedule,  the  developers  could  not  slip  the  date  for  decision  on  long-lead  items,  a 
date  initially  planned  to  succeed  the  product-verification  milestone.  Following  the  GE 
schedule  slips,  the  SPO  approved  long-lead  procurement  before  final  product  verification  was 
complete.  This  decision  potentially  increased  concurrency  between  development  and 

"*UnpubliHhe(i  history  report  of  the  Propulsion  SPO  Strutogit  Enginos  I^ogram  Offlco,  July- 
Uucember  I'JHJ;  “PllO-GE-lOO  Progrum  Hubinoss/Technicul  Uoviow,"  ASD  Deputy  lor  Propulsion, 
unpublirlied  briefing  cliurts,  Wright-Putlorsoii  Ait  Fnreu  Duse,  11  April  1984;  J.  K.  Nelson,  “CoinmunJ 
AssossmoiU  Hoview,"  unpublished  briofing  chi>"’.-  luty  for  Propulsion,  Wright-Putterson  Air  Forco 
Huso,  U)  April  liiH4i  unpublishod  history  riipori  oi  tlie  Propttlslon  8PO  Knginoeriiig  Offico,  Junuury- 
Juno  1984. 
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production,  something  the  four-step  process  sought  to  reduce.  But  sufficient  slack  allowed  a 
decision  without  a  significant  increase  in  risk.  Product  verification  occurs  over  a  long  time, 
and  the  items  most  critical  to  long-lead  procurement  had  been  verified  early  enough  to  allow 
a  long-lead  decision  without  delay. 

Slack  in  the  product  design  allowed  satisfactory  adaptation  to  another  unexpected 
event.  Late  in  the  development,  testers  identified  a  problem  with  “screech”  in  the  FllO 
augmentor,  a  dynamic  oscillation  of  flame  that,  in  a  small  part  of  the  flight  envelope,  could 
break  spraybars  in  the  augmentor,  cause  engine  fires,  and  endanger  the  engine  and  its 
aircraft.  Pilots  rarely  fly  in  the  affected  region,  but  consultations  with  future  users  in  the 
Tactical  Air  Command  (TAC)  determined  that  screech  remained  important  to  the  engine 
specifications  when  engines  were  deployed. 

The  developers  employed  a  three-part  response  to  this  unexpected  problem.  First, 
they  cut  the  air/fuel  ratio  in  the  augmentor,  reducing  thrust  somev/hat.  Second,  they  then 
reoptimized  the  operation  of  the  engine.  Test  data  suggested  that  sufficient  slack  exi.sted  in 
the  design  so  that  they  could  reduce  the  stall  margin  by  increasing  the  pressure  ratio  in  the 
compressor  to  compensate  for  some  of  the  loss  of  thrust  in  the  augmentor.  Third,  they 
developed  a  refinement  in  the  augmentor  spraybar.  This  third  step  took  a  great  deal  of  effort 
and,  in  fact,  continued  beyond  the  deployment  date  for  engines.  While  this  was  happening, 
the  Propulsion  SPO  continued  to  work  with  TAC  to  ensure  that  performance  changes 
associated  with  thsse  changes  were  satisfactory  from  the  user’s  point  of  view. 

In  the  end,  it  is  unclear  why  the  development  process  did  not  catch  this  problem 
earlier.  But  when  it  was  identified,  the  developers  used  an  orderly,  flexible  system  to 
respond  in  a  way  that  satisfied  the  final  engine  users. 

The  FlOO  DEEC/pump/valve/EMS  full-scale  development  that  pelded  the  -220  was  a 
much  more  complex  undertaking  than  the  FllO-GE-100  full-scale  development.  Here,  P&W 
and  the  SPO  coordinated  three  different  contracts  and  funding  sources:  the  full-scale 
development  itself,  the  supporting  CIP,  and  the  continuing  EMDP.  Each  had  well-defined 
tasks  that,  in  the  end,  created  the  capability  required  to  draw  up  specifications  for  cmd  verify 
the  production  version  of  the  -220,  The  success  of  this  effort  clearly  illustrated  the  synergy 
between  SPO  and  contractor  end  their  proficiency  in  working  with  several  programs  to 
produce  a  final  product  that  was  delivered  to  the  field  six  months  boforo  expected. 

Summary 

The  Propulsion  SPO  is  ditfeient  from  mosf  SPOs  boenune  it  remain.s  in  place  ns 
developna'nt.  and  production  programs  pas.s  through  it.  Tliorefure,  the  SPO  cun  build 
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experience  and  capability  in  a  well-designed  and  familiar  decision-making  environment.  The 
tradition  that  exists  today  was  still  young  when  the  FlOO-PW-220  and  FllO-GE-100 
developments  occurred,  but  important  elements  of  it  were  in  place.  The  managers  of  these 
developments  were  experienced  and  familiar  with  the  underlying  technology.  The  basic 
matrix  was  in  place  to  support  several  developments  at  once.  All  the  standard  procedures 
now  used  by  the  SPO  to  explain  its  longer  term  success  were  in  place.  With  experienced 
managers,  the  SPO  was  able  to  use  those  procedures  to  deal  confidently  with  unexpected 
events  in  a  flexible  and  often  creative  way.  With  its  contractors,  it  developed  creative  ways 
to  maintain  funding  for  their  efforts.  It  used  close  oversight  and  good  communication  to 
manage  the  developments  in  their  riskiest,  formative  days.  It  reacted  smoothly  to  test 
failures  with  alternative  technologies  and  development  changes  that  ultimately  enabled  the 
developments  to  meet  the  expected  performance  specifications  on  time  and  without  a  cost 
overrun. 

CONTRACTS 

Much  of  the  discussion  of  risk  management  in  acquisition  policy  has  emphasized  the 
importance  of  contracts,  and  contract  type  in  particular.  Contracts  did  play  a  key  role  in  the 
management  of  risk  in  the  two  developments,  but  not  necessarily  in  the  way  that  we  might 
expect  based  on  the  traditional  view  of  acquisition  policy.  Contract  type  is  only  one  of  several 
aspects  of  risk  management  in  the  contracts  used  to  implement  these  developments.  The 
writing  of  the  statements  of  work  and  special  clauses  in  the  development  contracts  and  in  the 
warranties  in  the  follow-on  production  contracts  was  also  important. 

Contract  Type  and  the  Statement  of  Work 

The  EMDP  and  full-scale-development  contracts  for  the  engine  developments  were  all 
firm-fixed-price  contracts.  The  FlOO  CIP  used  cost-plus-aw'ard-fee  contracts  throughout 
these  developments. The  common  wnsdom  today  strongly  frowns  on  the  use  of  firm-fixed- 
price  contracts  in  development  efforts,  although  they  appear  to  have  worked  satisfactorily  in 
the  two  developments.  In  fact,  the  use  of  firm-iixed-price  contracts  per  se  does  not  appear  to 
have  been  the  key  contracting  issue  in  these  developments. 

Most  observers  today  now  reject  firm-fixed-price  contracts  for  development  programs 
becauue  of  the  risk  involved.  It  is  extremely  difficult  to  predict  precisely  whut  a  development 
program  will  yield  or  whut  the  effort  will  cost.  Hence,  observers  often  argue  that  specifying  a 

'‘!U1''U)()  ('ll'  Incroiiicnt  VIll  in  1‘JHl  usud  rirni-fixiHi-prict'  lovol-ol'-cn'ort  tcr/iia  for  its  fllglit- 
tosling  ta.sks.  Those  torni.s  pliiyud  a  very  liiniUid  rolu  in  llv'  diivulopiaonls  wo  oxuinino  huro.  VSiW 
strongly  rosistod  llio  use  of  a  fii  iu-llxed-prieo  approach  imu  It  was  ilropisal  allor  this  contract. 
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firm  fixed  price  typically  imposea  unacceptable  risk  on  the  contractor.  The  contractors  in 
these  developments  did  not  balk  at  using  firm-fixed-price  contracts,  partly  because  of  the  low 
expected  technological  risk  in  these  developments.  But  a  contract  type  is  only  one  of  several 
elements  that  determine  the  risk  associated  with  a  contract. 

At  least  as  important,  the  goals  of  these  developments  were  stated  in  terms  that 
controlled  the  risks  involved  in  the  developments.  The  EMDPs  especially  placed  very  heavy 
emphasis  on  how  development  would  proceed,  rather  than  on  what  it  would  produce.  The 
statements  of  work  for  both  the  FlOO  and  the  FlOl  DFE  EMDPs  gave  substantial  attention 
to  levels  of  inputs  expected — for  example,  numbers  of  test  engines  and  numbers  of  hours  or 
flights  for  testing.  Final  system  requirements  were  more  clearly  stated  in  the  FSD  contracts; 
but  even  there,  the  development  process  received  considerably  more  attention  than  the 
development  product.  All  contracts  included  an  order-of-precetlence  clause  that  essentially 
stated  which  parts  of  the  contract  took  priority  when  different  parts  conflicted  with  one 
another.  In  that  clause,  the  contracts  gave  specifications  lower  prioiity  than  anything  else, 
effectively  emphasizing  the  way  the  development  was  conducted  over  what  it  produced  when 
questions  arose.  This  emphasis  is  typical  of  development  contracts;  specifications  typically 
receive  much  greater  relative  emphasis  in  production  contracts. 

Even  when  a  final  deliverable  was  defined,  the  terms  were  olton  broad.  For  example, 
the  FlOl  DFE  EMDP  envisioned  “a  derivative  design  of  the  FlOl  engine,  using  the  FlOl  core 
engine  design  to  the  ma.vimum  extent  p.issiblc.  The  engine  design  shall  be  satisfactory  for 
use  in  fighter  aircraft  applications. .  .  .  The  engine  shall  be  capable  of  producing  at  least 
28,000  pounds  of  thrust  at  sea  level  static  standard  day  conditions,  The  engine  design  and 
performance  will  conform  to  the  detailed  requirements  of  the  Engine  Specification;"  the 
contract  indicates  clsewlierc  that  the  convractor  will  provide  this  specification  us  part  of  tho 
development  effort..^'  The  FlOO  EMDP  act  a  long  list  of  development  goals  and  then  stated 
that  "These  Progi'am  Goals  are  for  planning  and  manugemi'nt  purposcH  and  are  not  to  be 
considered  criteria  for  successful  contract  completion.”^''  Discussions  of  the  development 
tasks  (as  opposed  to  products)  that  dominate  these  contracts  are  equally  general. 

The  risk  presented  in  u  devt.-lopmont  contract  depends  at  least  us  much  on  tin;  way  tin; 
statemunl  of  work  i.s  dtd’incd  us  un  the  contract  typo  itself.  01),servorH  suggest  tliat  tlo' 
statements  of  work  used  in  the  engine  duvelopmonts  iiro  not  unuiiual.  Tlsfy  r(illt!('.t  an 

-*L’  S.  y\ir  I'orcc  Contruci  l''.'l,'iGri7-7'J-tN017tJ,  Mudilicalion  I’ZODOl),  lOJuly  11171),  AiUidinaaiL  1, 
j))).  2,  ‘l-f) 

-'^U.S.  Air  I'orcii  (loMi.rucI-  l'','ia(!,')7-7ti-(MH)‘l  1,  Modllicatioo  I’/OOOl,  (i  May  AUaclniiool  1, 
pp.  'I'alih' .'XccrplH  Honii' Ilf  thrsa  goalit. 
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important  feature  of  a  relational  contract,  the  importance  of  watching  the  progress  of  the 
work  closely  in  a  relationship  that  is  expected  to  persist  over  the  long  term.  All  the 
development  contracts  .allow  the  Air  Force  to  remain  intimately  involved  in  the  process  of 
development  itself  They  provide  access  that  ensures  that  the  Air  Force  will  learn  about 
problems  quickly  as  the  developments  proceed  and  that  the  .'Vir  Force  can  react  quickly  to 
surprises  to  control  their  effect  on  the  program  as  a  whole.  Participants  in  the  development 
emphasize  that  this  monitoring  function  played  a  larger  role  in  these  developments  than  the 
general  type  of  contract.  And,  as  noted  above,  the  presence  of  competition  simplified  this 
monitoring  role:  It  made  it  harder  for  either  contractor  to  exploit  the  general  language  used 
to  specify  the  statements  of  work  in  the  contracts. 

Some  students  of  contracting  suggest  that  the  difference  between  fixed-price-  and  cost- 
plus-type  contracts  is  overstated  because  contractors  can  use  contract  modifications  in  a 
firm-fixed-price  contract  to  recover  their  costs  if  they  are  higher  than  expected.  All  the 
engine  contracts  experienced  substantial  contract  changes  over  their  lifetimes.  Without  a 
detailed  audit  of  those  changes,  we  cannot  state  that  they  were  not  used  to  cover  unexpected 
costs  earlier  in  a  program.  But  numerous  auditors  who  examined  the  programs  found  no 
evidence  of  such  a  problem  On  the  face  of  them,  the  modifications  of  these  contracts  appear 
to  have  been  driven  by  reasonable  changes  in  work  scope  required  to  let  the  developments 
continue  in  fruitful  directions.  The  number  of  changes  and  the  ease  with  which  they  were 
implemented  appear  to  reflect  a  well-functioning  system  that  promoted  the  kind  of  flexibility 
the  changes  made  possible.  Perhaps  more  to  the  point,  GE  and  P&W  have  both  shown  a 
strong  preference  for  cost-based  contracting  when  offered  the  choice,  which  suggests  that 
both  experience  a  greater  degree  of  freedom  under  cost-based  contracts  than  under  fixed- 
price  contracts. 

The  simultaneous  use  of  cost-plus  and  fixed-price  contracts  in  the  FlOO-PW-220 
development  created  a  small  problem  in  that  development.  Because  CIP  tasks  cannot  be 
known  far  in  advance,  P&W  traditionally  accumulated  an  inventory  of  components  under  the 
CIP  that  could  support  a  wide  range  of  tasks  in  the  future.  The  CIP  is  flexible  enough  to 
allow  this  practice  and,  although  it  does  impose  perhaps  unexpected  inventory  costs  on  the 
U.S.  government,  it  contributes  to  the  flexibility  that  the  CIP  is  meant  to  maintaiin.  W'hen 
paired  with  fixed-price  contracts,  however,  accumulation  of  inventory  under  CIP  can  allow 
rr  '’^xibility  than  intended.  All  of  the  contracts  in  the  two  developments  were  fairly 
flexi  c  In  particular,  they  allowed  certain  transfers  of  government-owned  equipment 
among  specified,  closely  related  programs.  Under  such  circumstances,  a  strong  temptation 
existed  to  buy  components  under  the  cost-based  CIP,  where  the  contractor  could  recover  the 


costs  of  these  assets  from  the  government,  and  use  them  in  the  EMDP  or  full-scale- 
development  program,  both  firm-fixed-price,  and  for  which  the  contractor  could  not  recover 
costs  from  the  government,  The  Air  Force  had  to  tighten  control  on  the  acquisition  and  use 
of  inventories  under  the  CIP  to  avoid  this  kind  of  problem.  The  problem  is  a  direct  result  of 
the  type  of  contracts  chosen  for  the  development  programs. 

Special  Clauses 

The  contract  type  and  statement  of  work  are  not  the  only  parts  of  a  contract  that 
provide  information  on  the  risk  in  a  development  program  and  the  developers’  approach  to  it. 
Special  clauses  pro\ide  additional  information.^^ 

In  the  subject  fixed-price  contracts,  the  most  important  special  clause  was  the 
limitation-of-govemment-obligation  (LOGO)  clause.  Each  contract  had  it.^'*  The  clause  was 
required  because,  unlike  production  contracts,  uevelopnienL  contracts  need  not  be  fully 
funded.  This  clause  limits  the  government’s  obligation  under  a  contract  to  the  level  of 
funding  explicitly  obligated  to  the  contract,  thereby  eliminating  the  risk  that  the  Air  Force 
might  be  required  to  pay  for  services  with  funds  that  it  does  not  have.  In  all  of  these 
contracts,  the  most  common  contract  modification  increased  the  level  of  funding  available, 
effectively  increasing  the  amount  of  work  that  could  be  performed  under  the  contract.  Over 
their  lifetimes,  the  developments  were  rarely  fiilly  funded.  As  a  result,  the  contracts  often 
used  the  level  of  funds  obligated  instead  of  the  contract  price  as  the  best  measure  of  the 
contract  size.  This  clause  clearly  limited  the  Air  Force’s  risk  in  the  event  that  insufficient 
funds  were  aveiilable.  In  the  end,  all  the  contracts  were  fully  funded  and,  with  the  exception 
of  later  CIP  contracts,  the  availability  of  funding  had  no  effect  on  the  actual  activities  of 
development.  The  developers  had  an  understanding  with  the  Air  Force  comptroller,  often  in 
writing,  about  when  funds  would  arrive,  and  they  generally  arrived  as  scheduled. 

Systematic  delays  in  annual  congressional  funding  decisions  for  the  program  made  this 
understanding  more  difficult  to  sustain  in  later  CIP  contracts. 

The  FlOO  DEEC.'pump/valve/EMS  full-scale-development  contract  provided  another 
clause  to  deal  with  more  fundamental  uncertainty  about  funding.  It  listed  five  options  with 
not-to-exceed  quantities,  exercise  dates,  and  incremental  statements  of  work.^^  As  funding 
became  available,  the  Air  Force  exercised  these  options  and  effectively  expanded  the  scope  of 

^^For  a  discussion  of  use  of  special  clauses  to  promote  fairly  detailed  risk  assessment  in  Air 
Force  contracts,  see  Bodilly,  Camm,  and  Pei,  1991. 

2-*Cost-based  contracts  such  as  the  CIP  contracts  used  an  equivalent  device  callrd  a  limitation- 
of-funds  clause. 

25U.S.  Air  Force  Contract  F33657-82-C-2199,  Section  H,  Clause  ]  1,  pp.  21B-210  and 
Attachment  1,  Addenda  1-5. 
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ihe  effort.  Merely  structuring  such  options  helps  structure  the  development  effort  in  a  way 
that  accommodates  uncertainty  about  funding;  exercising  them  or  not  provides  a  well- 
structured  way  to  share  risk  about  funding.  The  Propulsion  SPO  has  used  such  an  approach 
in  other  development  contracts.  The  CIP  resolution-of-problems  clause  offers  a  standardized 
means  of  managing  such  risk  within  the  CIP  itself. 

To  manage  another  form  of  major  risk,  the  FlOO-GE-100  full-scale-development 
contract  had  a  clause  saying  that  “the  prices  and  the  delivery  provisions  of  this  contract  do 
not  include  provisions  for  major  repair,  replacement  or  retesting  of  the  engine,  components, 
and/or  special  tost  equipment  due  to  loss,  destruction,  or  damage  incurred  as  the  result  of  a 
catastrophic  failure  during  the  testing  ...  [or  for] .  .  .  any  individual  repair,  replacer.ient,  or 
retesting  of  which  the  agreed  allocable  costs  shall  exceed  $1,000,000  and  which  are  not 
compensated  by  insurance.  .  .  .  Any  such  major  repair,  replacement  or  retesting  shall  be  the 
subject  of  separate  negotiation."  At  this  stage  in  development,  such  a  risk  was  perhaps  the 
largest  risk  internal  to  the  development  effort  itself  This  clause  significantly  reduced  its 
importance  in  the  context  of  a  finm-fixed-pnce  contract. 

Other  specialized  clauses  dealt  vdth  specific  concerns  in  each  contract.  For  example, 
while  a  legal  issue  of  how  to  allocate  a  part,icular  form  of  corporate  overhead  costs  remained 
before  an  arbiter,  the  FlOO  DEEC/pump/valve/EMS  contract  used  a  clause  to  bound  the 
outcomes  that  were  mutually  acceptable  to  P&W  and  the  Air  Force. The  FllO-GE-100  full- 
scale-development  contract  coutained  a  similcm  clause  to  characterize  the  treatment  of 
independent  research  and  development,  bid  and  proposal,  and  related  fuel  and  tax  costs. 

Warranties 

Final  performance  on  each  full-scale-development  effort  was  judged  by  comparing 
actual  engine  specifications  with  the  over  one  thousand  pages  of  specifications  in  the  contract 
that  described  the  engine  contracted  for.  A  v/arranty  offers  one  way  to  reduce  these 
thousand-plus  pages  to  a  few  key  criteria  to  facilitate  ongoing  oversight  of  the  engine  and  its 
promised  performance  after  it  is  deployed.  Knowing  that  production  contracts  will  contain 
such  contractual  devices  potentially  motivates  developers  to  adjust  their  designs  in 
anticipation. 

Formal  warranties  did  not  exist  until  the  production  contracts  became  effective.  In 
this  sense,  their  effect  on  development  is  very  much  like  that  of  the  formal  competition 

2CU.S.  Air  Force  Contract  F33657-82-C-2199.  30  September  1982,  Section  H,  Clause  12,  p.  21D. 

2'U.S.  Air  Force  Contract  F33657-82-C-0257,  15  November  1082,  Section  H,  Clause  23,  p.  38. 
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embodied  in  these  same  contracts;  although  warranties  did  not  exist  earlier,  ebservers  and 
participants  widely  anticipated  their  use  early  in  the  developments. 

GE  in  particular  expected  to  use  warranties  to  promote  its  entry  into  the  fighter 
market.  The  Air  Force  was  most  concerned  about  operability  and  the  long-term  cost  of 
engine  ownership.  GE  believed  that  it  could  best  demonstrate  its  responsiveness  to  these 
concerns  by  offering  warranties  that  made  guarantees  about  these  concerns.  Hence,  early  in 
the  development  of  the  FlOl  DFE,  GE  developed  an  operating-and-support-cost  model,  based 
on  its  existing  commercial  engines,  to  estimate  the  likely  costs  of  the  engine  that  resulted 
from  the  DFE  development.  It  used  this  model,  which  incorporated  many  of  the  new 
techniques  emerging  under  ENSIP,  to  estimate  a  reasonable  price  for  a  warranty  at  that 
time. 

The  request  for  proposal  for  the  first  production  competition,  released  in  December 
1982,  included  explicit  provision  for  a  warranty  that  the  Air  Force  expected  to^ 

•  motivate/  give  incentive  to  the  contractor 

—  influence  design,  manufacturing,  test,  and  support  activities 

—  deliver  a  quality  product  initially  and  consistently 

—  stabilize  the  engine  support  system  early 

—  retain  thrust,  operability,  availability,  and  reliability 

•  satisfy  the  1983  3,O00-TAC-cycle  warranty  statute.*** 

Figure  5.1  summarizes  the  terms  of  the  proposed  warranty.  The  Air  Force  would  have  the 
option  of  repairing  failures  under  the  first  two  parts  and  receiving  compensation  from  the 
contractor  based  on  a  specified  schedule  of  costs.  The  first  portion  of  Part  I  is  essentially  a 
standard  warranty  covering  the  early  part  of  the  engines’  lives.  The  remaining  parts  look 
farther  into  the  future,  aiming  directly  at  the  development  goals  of  ensuring  operability, 
durability,  and  perfortr.ance  retention. 

The  Air  Force  released  these  terms  at  about  the  same  time  that  the  full-scale 
developments  began  As  indicated  above,  the  Air  Force  expected  the  warranties  to  influence 
the  ftesipn  of  the  engines  What  influence  did  they,  and  earlier  expectations  about 
warranties,  have  on  the  deveiupmoiit.s  themselves? 

28U.S.  Air  Force,  Aeronautical  Systems  Division  Deputy  for  Propulsion,  1984. 

29SeLtion  797  of  the  FT  1983  DoD  Appropriation  Act  states  that  no  money  shall  go  to  an 
•olte  mate  or  new  model  fighter  entrinr.  without  n  warranty  cni.irop'aoir"  .3,C0U  iuctical  cycles. 
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Part  I:  To  assure  quality,  avoid  infant  mortality,  assure  consistency,  retain  performance 
For  3  years  or  1,000  engine  flight  hours  (EFH): 


engines,  modules,  components,  parts 

defects  in  materials  and  workmanship 

parts  and  labor 

unusable/unserviceable  conditions 

parts 

operation  outside  tech  order  limits 

parts 

support  equipment  defects  in  materials  and  workmanship  or  function 

or  operations  outside  required  limits 

parts  and  labor 

For  3,000  TAG  cycles  or  8  years; 

performance  retention 

more  than  98%  intermediate  thrust 

parts  and  labor 

less  than  105%  intermediate  SFC 

parts  and  labor 

Part  II:  To  emphasize  high  cost/maintenance  dnvers,  assure  quality,  and  avoid  major 

support  destabilizers 

For  3,000  TAG  cycles  or  8  years: 

combustor  and  high-pressure  turbine 

unusable/unserviceable  conditions 

parts 

operation  outside  tech  order  limits 

parts 

Part  III:  To  assure  long-term  stability  in  fleet  support,  incentivize  contractor,  and 
share  risk 

3,000  TAG  cycle  demonstration: 

engine  fleetwide  combined  UER  and  SEU'IOOO  EFH 
contractor-designated  rate  for  F-15,  F-16 
first  demonstration  period,  1987-88,  with  assessment  in  1989 
annual  demonstrations  through  1994,  with  assessment  a  year  following 
contract  adjustment  for  rate  better/worse:  ±  $25,000/removal 

SOURCE:  U.S.  Air  Force,  Aeronautical  Systems  Diviaion  Deputy  for  Propulsion,  1984. 


Figure  5.1 — Summary  Terms  of  Warranty  Called  for  in  Request  for  Proposals 
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Participants  in  the  developments  discount  the  effects  of  the  warranty  on  the 
developments  themselves.  They  emphasize  the  role  of  warranties  to  define  and  explicitly 
share  risk  after  the  engines  reached  the  field.  The  prices  demanded  for  warranties  provided 
useful  information  on  the  maintenance  costs  that  each  developer  expected  for  its  design, 
given  the  design  chosen.  Participants  believe  that  the  designs  of  the  engines  that  actually 
reached  the  field  would  have  been  of  about  the  same  quality  with  or  without  the  warranties. 

We  cannot  prove  or  disprove  this  assertion  by  looking  at  only  one  case.  But  it  seems  to 
defy  common  sense,  particularly  if  wc  view  the  warranties  in  the  context  of  competition.  GE 
effectively  proposed  a  warranty  as  a  way  to  market  its  new  capability.  Given  the  Air  Force's 
priorities  in  the  competition,  P&W  had  to  recognize  that,  to  be  competitive,  this  warranty 
would  enforce  a  similar  requirement  on  P&W.  If  P&W  was  to  meet  GE’s  threat  effectively,  it 
had  to  design  an  engine  good  enough  to  allow  P&W  to  offer  a  competitive  warranty.  And 
once  committed  to  a  competition  defined  in  part  by  warranties,  both  developers  had 
incentives  to  design  engines  compatible  with  such  a  competition.  Perhaps  the  exact  terms  of 
the  warranty  were  finalized  too  late  to  influence  the  exact  design  of  the  two  engines. 

Perhaps  the  role  of  competition  in  this  scenario  is  so  large  as  to  overwhelm  the  role  of 
warranties.  Most  observers  agree  that  we  cannot  separate  the  warranties  from  the 
competition — the  competition  could  not  have  been  as  successful  w'ithout  warranties  and  the 
Air  Force  could  not  ultimately  have  negotiated  as  satisfactory  terms  on  the  warranties 
without  competition.^®  The  issue  bears  additional  attention. 

Summary 

Discussions  of  ri.sk  management  in  acquisition  contracts  tend  to  emphasize  contract 
type.  WTiile  contract  fyqie  is  not  unimportant,  other  aspects  of  contracts  also  play  an 
important  role.  The  development  contracts  we  examine  used  statements  of  work  that 
addressed  not  just  the  final  product  of  development,  but  the  process  of  development,  focusing 
primarily  on  development  tasks  that  are  easier  to  understand  in  advance  than  development 
products.  They  also  employed  special  clauses  to  deal  with  major  risks  related  to  funding  and 
test  events.  Perhaps  as  a  result,  P&W  and  GE  appear  to  have  had  little  objection  to 
conducting  development  activities  under  lirm-fixed-price  contracts.  The  low  inherent  risk  in 
the  developments,  of  course,  was  also  a  factor.  The  production  contracts  that  followed  the 

3^In  fact,  P&W  demanded  a  very  high  price  for  its  warranty  in  the  first  competition  if  the  Air 
Force  split  its  buy.  A  successful  competition  effectively  increased  P&W^’s  risks,  and  P&W  found  one 
way  to  express  that  increase  by  demanding  a  higher  price  for  bearing  the  risk  of  the  warranty  if 
competition  continued  than  if  the  Air  Force  foreclosed  the  GE  alternative.  The  Congress  refused  to  pay 
this  high  price  following  '^he  first  production  competition;  P&W  relented,  and  the  price  of  P&Ws 
warranty  closely  matched  GE's  price  in  future  competitions. 
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developments  included  detailed  warranties  designed  to  define  and  manage  risk  following 
deployment.  Whether  anticipation  of  those  warranties  helped  define  and  manage  risk  during 
development  is  unclear. 

SECTION  SUMMARY 

Participants  in  the  derivative  engine  developments  did  not  ever  perceive  them  as 
presenting  high  risks.  As  derivative  engines,  the  FlOO-PW-220  and  FllO-GE-100  presented 
low  technological  risks.  And  because  the  developments  were  small  a*id  unJikely  to  affect  any 
other  higher  profile  programs,  officials  above  the  Air  Force  imposed  few  risks  on  them  other 
than  balking  initially  about  funding. 

Such  low  risks  significantly  contributed  to  the  success  of  these  developments.  But 
other  factors  contributed  as  well.  Active,  if  informal,  competition  significantly  increased  the 
Air  Force’s  power  in  its  long-term  relationships  with  P&W  and  GE.  Competition  increased 
P&W  i  nd  GE's  responsiveness,  simplified  the  Air  Force’s  need  to  monitor  them,  and  made  it 
easier  for  the  Air  Force  to  respond  flexibly  to  unexpected  events  during  the  developments. 

An  experienced  SPO,  with  well-developed  standard  procedures  and  a  rapidly  expanding 
formal  doctrine  on  risk  management  also  contributed  to  the  developments’  flexibility.  The 
SPO  and  its  contractors  repeatedly  managed  unexpected  events  in  the  developments  with 
skill  and  confidence.  Low  risk  facilitated  the  use  of  finn-fixed-price  contracts  in  the 
developments,  but  other  contractual  factors  enriched  those  contracts’  ability  to  reduce  and 
share  risk.  The  statements  of  work  reduced  risk  by  emphasizing  process  over  product; 
special  clauses  managed  concerns  about  specific  uncertainties  about  funding,  the  cost  of 
catastrophic  test  failures,  and  other  factors.  Warranties  clearly  helped  manage  risk  during 
the  production  and  deployment  of  engines  that  the  development  programs  created;  their 
effects  on  risk  management  during  the  developments  are  less  clear. 
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6.  CONCLUSIONS 


We  conducted  this  study  as  one  of  seven  case  studies  designed,  when  taken  together, 
to  yield  useful  insights  about  Air  Force  development  practices  and  the  assessment  and 
management  of  risk  in  those  practices.  We  must  be  cautious  drawing  firm  conclusions  from 
the  one  case  presented  here.  Nonetheless,  this  case  develops  suggestive  information  that  we 
can  present  in  the  form  of  six  hypotheses.  These  hypotheses  suggest  information  to  look  for 
in  the  other  cases  as  well  as  ideas  for  future  policy  and  policy  analysis. 

1.  Derivative  development  can  reduce  risk  in  limited  circumstances.  The  Air 
Force  uses  derivative  engine  development  when  they  allow  such  development  to  achieve  its 
goals.  But  derivative  developments  do  not  allow  large  advances  in  system  performance, 
broadly  conceived.  Large  advances  require  more  traditional,  centerline  engine  developments, 
which  inherently  entail  more  technological  risk  than  that  observed  here. 

This  view  of  derivative  development  finesses  the  question  of  whether  larger  advances 
are  worth  the  risk  or  whether  the  current  mix  of  derivative  and  centerline  developments 
found  in  the  Air  Force  is  appropriate  today  or  would  be  appropriate  in  the  rapidly  changing 
environment  of  the  near  future.  We  can  expect  that  environment  to  increase  the  external 
risk  experienced  in  most  developments;  greater  emphasis  on  derivative  development  may 
offer  one  way  to  offset  that  risk  increase,  We  cannot  say  more  about  this  possibility  without 
giving  the  question  additional  attention. 

2.  Continuity  in  SPO  management  significantly  enhances  risk  management 

As  a  standing  SPO  that  does  not  come  auid  go  with  individual  development  programs,  the 
Propulsion  3PO  has  accumulated  experience,  doctrine,  procedures,  and  knowledge  about  its 
contractors,  all  of  which  contribute  significantly  to  its  ability  to  anticipate  and  manage 
problems.  These  sources  of  order  and  stability,  perhaps  ironically,  gave  the  SPO  the  skill 
and  confidence  required  to  react  flexibly  and  creatively  to  unexpected  events  in  the  subject 
developments.  Analogous  statements  apply  to  the  contractors  themselves. 

It  is,  of  course,  easy  to  say  that  more  continuity  is  better.  \\’hat  is  the  cost  of 
increasing  continuity?  For  engines,  continuity  is  possible  because  engines  are  all  managed 
by  a  standing,  collective  staff  instead  of  being  placed  in  aircraft  3POs,  as  they  were  before 
the  Propulsion  SPO  formed  in  1977.  Total  system  management  of  new  aircraft  emphasizes 
the  importance  of  integrating  the  management  of  all  subsystems  of  an  aircraft  in  one  place. 
ASD  turned  back  toward  this  approach  with  the  B-2  development,  which  integrated 
development  of  the  FI  18  engine  with  development  of  the  airframe  to  facilitate  security  goals. 
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ASD  has  also  placed  responsibility  for  developing  the  engine  for  the  advanced  tactical  fighter 
(ATF)  with  the  airframe  SPO,  not  the  propulsion  SPO,  to  ensure  proper  integration. 

Placing  all  engine  development  in  one  SPO  rriakes  it  more  'difficult  to  manage  system 
in:;egration  and  security,  but  giving  these  goals  too  much  emphasis  poses  a  serious  risk.  The 
Air  Force  formed  the  Propulsion  SPO  to  solve  some  serious  problems.  The  story  of  the 
subject  developments  is  part  of  the  story  of  how  the  SPO  solved  those  problems.  With  the 
problems  apparently  solved,  it  may  be  easy  to  forget  their  importance  and  their  solution.  An 
important  lesson  of  these  developments  is  that  continuity  in  a  SPO  makes  it  easier  for 
lessons  learned  not  to  be  lost,  including  lessons  about  continuity. 

3.  Formal  risk  assessment  plays  a  limited  but  important  role  in  general  risk 
management.  New  concepts  and  methods  developed  in  the  1970s,  including  the  accelerated 
mission  test,  the  four-step  development  process,  and  the  Engine  Structural  Integrity 
Program,  gave  the  developers  of  the  two  engines  tools  for  managing  numerous  important 
risks.  Without  those  tools,  the  developers  could  not  have  achieved  the  durability  and 
supportability  goals  sought  for  the  engines.  But  such  methods  do  not  help  managers  deal 
with  broader,  less-defned  nsks.  Here,  there  remains  no  substitute  for  the  judgment  of 
experienced  managers  familiar  with  the  routine  management  of  risk  in  system 
developments. 

Formal  risk  assessment  is  possible  only  when  the  risks  involved  are  simple  enough  to 
be  viewed  in  a  simple  analytic  framework.  And  even  then,  the  results  of  the  assessment  are 
no  better  than  the  inputs  to  the  framework.  The  value  of  the  methods  mentioned  above  is 
twofold:  they  reflect  major  new  insights  into  the  rea"''"s  why  engines  and  their  components 
fail  and  the  way  in  which  they  fail;  and  the  Air  For  ;•  -  its  contractors  have  been  able  to 
collect  fairly  objective  empirical  data  to  implement  those  methods.  There  will  always  be 
more  room  for  improvement  in  our  understanding  of  failure  modes  and  of  the  data  we  use  to 
quantify  them,  and  future  analysis  should  pursue  both.  But  neither  will  ever  replace  the 
basic  intuition  of  a  manager  acting  in  an  unstructured  setting.  Making  a  decision  in  such  a 
setting  is  essentially  the  raison  d’etre  for  a  manager,  and  no  amount  of  formal  risk 
assessment  will  replace  the  manager’s  role.  That  said,  we  must  accept  that  managers  react 
to  their  incentives.  And  the  problem  of  proriding  managers  with  incentives  to  “do  the  right 
thing”  should  benefit  as  much  from  additional  analysis  as  from  more  precise  questions  raised 
by  formal  risk  assessment. 

4.  Higher  level  interest  in  a  program  increases  some  program  risks  and 
reduces  others.  The  general  absence  of  higher  level  interest  in  the  developments  allowed 
their  immediate  managers  to  set  realistic  performance,  schedule,  and  cost  goals  and 
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ultimately  to  meet  those  goals.  In  this  sense,  the  lack  of  interest  reduced  the  external  risk 
experienced  in  the  developments.  But  a  lack  of  interest  also  left  the  developments  without 
high-level  advocates  to  protect  the  programs’  funding,  increasing  external  risk.  Such  risks 
must  be  balanced. 

One  way  to  reduce  this  conflict  is  to  offer  oversight  well  informed  about  the  importance 
of  setting  and  achieving  realistic  goals.  Another  way  to  say  this  is  to  encourage  high-level 
policy  makers  to  believe  that,  if  a  developer's  omises  look  too  good  to  be  true,  they  probably 
are.  Encouraging  or  even  trying  to  force  the  developer  to  seek  unrealistic  goals  is  unlikely  to 
be  a  successful  strategy  in  the  long  run.  To  know  whether  promises  are  reasonable  or  not,  of 
course,  the  policy  maker  mu.'it  be  well  informed.  A  message  offered  over  and  over  by 
participants  in  the  developments  is  that  the  overseer’s  knowledge  is  critical  to  the  success  of 
a  development  because  it  is  extremely  difficult  to  create  incenti*  es  that  induce  a  developer  to 
be  fully  open. 

Perhaps  the  most  difficult  management  problem  in  a  development  results  from  the 
fact  that,  even  if  a  development  goes  poorly,  it  may  continually  seem  to  be  rational  to 
proceed.  If  the  tnie  path  of  the  development  had  been  known  from  the  beginning,  it  might 
not  have  been  rational  to  start.  But,  at  each  point  along  the  way,  what  is  past  is  sunk  and  it 
may  appear  that  future  benefits  exceed  future  costs.  Knowing  this,  a  contractor  has  a  strong 
incentive  to  get  the  government  committed  early  and  to  continue  ever  hopeful  that  things 
will  work  out.  The  Air  Force  presumably  has  similar  incentives  relative  to  the  Office  of  the 
Secretary  of  Defense  and  the  Congress.  Only  a  well-informed  overseer  can  anticipate  the 
problems  of  development  and  plan  against  them.  This  insight  helps  explain  the  Propulsion 
SPO’s  commitment  to  hands-on  management  and  suggests  that,  properly  understood,  this 
insight  has  much  broader  application. 

5.  Competition  redistributes  risk  in  a  deve.upnient  program.  Competition 
increases  the  risk  perceived  by  the  contractors  and  reduces  the  risk  perceived  by  the  Air 
Force.  At  any  point  in  time,  the  A'r  Force  can  change  the  market  shares  of  its  contractors, 
increasing  their  risks,  and  by  so  doing  can  reduce  its  dependence  on  the  poorer  performer, 
reducing  its  own  risk.  Competition  also  increases  the  risk  that  the  Air  Force  can  induce  its 
contractors  to  bear  at  a  given  cost.  These  changes  in  risk  result  from  a  basic  shift  in  relative 
power  that  increases  the  contractors'  responsiveness  and  reduces  the  Air  Force’s  cost  of 
monitoring  their  contract  compliance. 

Using  competition  to  get  these  benefits  is  not  costless.  Carrying  two  engines  through 
full-scale  development  essentially  doubled  the  cost  of  creating  the  technological  capability 
offered  by  the  derivative  engines.  Competition  may  lave  led  to  some  economies  during  tne 
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development  phase  that  kept  costs  from  fully  doubling,  but  few  would  argue  that  such 
economies  fully  offset  the  cost  of  developing  a  second  engine.  Even  all  the  added  benefits  of 
responsiveness  and  risk  shifting  during  development  are  unlikely  to  justify  such  additional 
costs.  The  real  benefits  come  over  the  life  of  the  production  of  the  engines.  Xo  good 
estimates  exist  for  the  benefits  of  competition  over  this  period.  In  fact,  the  usual  analytic 
view  of  competition,  which  does  not  examine  the  effects  of  power  shifting  in  small-number 
relationships,  has  no  widely  accepted  way  to  measure  such  benefits.  The  question  deserv'es 
more  attention. 

It  is  instructive  to  note  that,  after  two  competitions  touted  by  observers  and 
participants  in  the  Air  Force  acquisition  community  as  highly  successful,  the  Alternate 
Fighter  Engine  and  improved  perfonnance  engine  competitions,  the  Air  Force  rejected  the 
use  of  competition  in  its  next  major  engine  development,  ihe  full-scale  development  of  the 
engine  for  the  advanced  tactical  fighter  A;iparently  the  Air  Force  found  the  cost  of 
developing  a  second  engine  through  product  verid  cation  too  high  to  support  as  its  overall 
budget  falls.  Officials  appear  to  believe  that  this  cost  would  justify  itself  over  the  course  of 
the  full  production  run;  but  the  initial  cash  flow  expected  for  a  second  full-scale  development 
would  be  too  hard  to  justify  to  Congress  in  a  period  of  uncertainty  and  retrenchment, 

Perhaps  competition  for  derivat-ve  engines  was  so  successful  because  the  cost  of  developing  a 
second  engine  through  product  verification  was  low  enough  not  to  attract  undue  attention  to 
itself. 

6.  Other  contractual  terms  are  at  least  as  important  ns  contract  type  in  risk 
management.  The  choice  of  contract  type  affects  the  business  environment  for 
development;  contractors  clearly  prefer  a  cost-based  to  a  fixed-price  contract  if  both  types  are 
offered  with  similar  terms.  But  a  contract’s  statement  of  work  and  special  clauses  can  be 
WTitten  to  seften  the  risks  implied  by  a  fixed-price  contract,  pote-'tially  dominating  the  choice 
of  contract  type  in  their  effects  on  contractor  risk.  And  the  techniques  used  by  the  SPO  to 
monitor  contract  compliance  can  have  far  larger  effects  on  risk  management  than  the  formal 
terms  of  the  contract. 

These  conclusions  are  especially  apropos  in  development  contracts,  the  statements  of 
work  for  whicn  do  not  use  the  precision  found  in  a  typical  production  contract.  And  simple 
special  clauses  can  be  written  to  reduce  dramatically  the  risk  of  large  losses,  which  concern 
contractors  the  most.  Such  clauses  may  also  be  especially  important  in  the  near  future  as 
defense  budgets  shrink  and  the  number  of  prime  contractors  involved  in  defense  work 
shrinks.  Longer  term  relationships  between  the  government  and  individu-il  development 
contractors  could  well  become  more  important,  emphasizing  the  need  not  just  for  clearly 
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written  contracts  and  contract  law  but  for  a  well-informed  working  relationship  between  the 
Air  Force  and  its  contractors  that  sustains  the  increasingly  important  relational  contracts  in 
defense  system  management. 
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